This way the clearing can be skipped in case of some errors.

Fixes: Timeout (11sec -> 344ms)
Fixes: 14670/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PAF_VIDEO_fuzzer-5769534503387136

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>