Any of these buffers (for both H.264 and H.265) might reasonably be
parsed using the bitstream reader, so include padding on all of them.