Fixes high memory usage and prevents over allocation of the frames via
proper unref.

Can be checked as:
-hwaccel qsv -c:v h264_qsv -i ../h264-conformance/CANL2_Sony_E.jsv -c:v
h264_qsv -b:v 2000k -y qsv.mp4