This avoids unneeded operations and makes the code faster.

Fixes: Timeout
Fixes: 15724/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PAF_VIDEO_fuzzer-5750842205929472 (12sec -> 9sec)
Fixes: 17625/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PAF_VIDEO_fuzzer-5640515311108096 (16sec -> 4sec)

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>