This filter currently assumes that the input audio is continuous and
does some timestamps manipulation based on this assumption.

This is unnecessary if we are only converting the channel layout or the
sample format, without resampling. In such a case, just leave the
timestamps as they are.