Commit 97472199 authored by wm4's avatar wm4

avcodec/h264, videotoolbox: do not return invalid frames on failure

If videotoolbox_common_end_frame failed, then the AVFrame was returned
to the API user with the dummy buffer (in AVFrame.buf[0]) still set, and
the decode call indicating success.

These "half-set" AVFrames with dummy buffer are a videotoolbox specific
hack, because the decoder requires an allocated AVFrame for its internal
logic. Videotoolbox on the other hand allocates its frame itself
internally, and outputs it only on end_frame. At this point, the dummy
buffer is replaced with the real frame (unless decoding fails).
parent 8eadabf9
......@@ -515,6 +515,7 @@ void ff_h264_remove_all_refs(H264Context *h)
if (h->short_ref_count && !h->last_pic_for_ec.f->data[0]) {
ff_h264_unref_picture(h, &h->last_pic_for_ec);
if (h->short_ref[0]->f->buf[0])
ff_h264_ref_picture(h, &h->last_pic_for_ec, h->short_ref[0]);
}
......
......@@ -353,6 +353,8 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame)
AVVideotoolboxContext *videotoolbox = avctx->hwaccel_context;
VTContext *vtctx = avctx->internal->hwaccel_priv_data;
av_buffer_unref(&frame->buf[0]);
if (!videotoolbox->session || !vtctx->bitstream)
return AVERROR_INVALIDDATA;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment