There have been multiple bugs caused by inconsistencies here.

Based on an idea from Michael Niedermayer.

CC: libav-stable@libav.org