This will allow the decoder to return samples for the full packet, and it also
makes the decoded buffer pointers aligned.