This also allows for removing some of the buf_size checks and using the
sample count for some of the decoding loops.