It represents the relationship between them more naturally and will be
useful in the following commits.

Allows significantly more frames in fate-h264-attachment-631 to be
decoded.