• Aman Gupta's avatar
    avcodec/h264, videotoolbox: fix crash after VT decoder fails · b6eaa392
    Aman Gupta authored
    The way videotoolbox hooks in as a hwaccel is pretty hacky. The VT decode
    API is not invoked until end_frame(), so alloc_frame() returns a dummy
    frame with a 1-byte buffer. When end_frame() is eventually called, the
    dummy buffer is replaced with the actual decoded data from
    VTDecompressionSessionDecodeFrame().
    
    When the VT decoder fails, the frame returned to the h264 decoder from
    alloc_frame() remains invalid and should not be used. Before
    97472199, it was accidentally being
    returned all the way up to the API user. After that commit, the dummy
    frame was unref'd so the user received an error.
    
    However, since that commit, VT hwaccel failures started causing random
    segfaults in the h264 decoder. This happened more often on iOS where the
    VT implementation is more likely to throw errors on bitstream anomolies.
    A recent report of this issue can be see in
    http://ffmpeg.org/pipermail/libav-user/2016-November/009831.html
    
    The issue here is that the dummy frame is still referenced internally by the
    h264 decoder, as part of the reflist and cur_pic_ptr. Deallocating the
    frame causes assertions like this one to trip later on during decoding:
    
      Assertion h->cur_pic_ptr->f->buf[0] failed at src/libavcodec/h264_slice.c:1340
    
    With this commit, we leave the dummy 1-byte frame intact, but avoid returning it
    to the user.
    
    This reverts commit 97472199.
    Signed-off-by: 's avatarwm4 <nfxjfg@googlemail.com>
    b6eaa392
Name
Last commit
Last update
compat Loading commit data...
doc Loading commit data...
libavcodec Loading commit data...
libavdevice Loading commit data...
libavfilter Loading commit data...
libavformat Loading commit data...
libavresample Loading commit data...
libavutil Loading commit data...
libpostproc Loading commit data...
libswresample Loading commit data...
libswscale Loading commit data...
presets Loading commit data...
tests Loading commit data...
tools Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.travis.yml Loading commit data...
CONTRIBUTING.md Loading commit data...
COPYING.GPLv2 Loading commit data...
COPYING.GPLv3 Loading commit data...
COPYING.LGPLv2.1 Loading commit data...
COPYING.LGPLv3 Loading commit data...
CREDITS Loading commit data...
Changelog Loading commit data...
INSTALL.md Loading commit data...
LICENSE.md Loading commit data...
MAINTAINERS Loading commit data...
Makefile Loading commit data...
README.md Loading commit data...
RELEASE Loading commit data...
arch.mak Loading commit data...
cmdutils.c Loading commit data...
cmdutils.h Loading commit data...
cmdutils_common_opts.h Loading commit data...
cmdutils_opencl.c Loading commit data...
common.mak Loading commit data...
configure Loading commit data...
ffmpeg.c Loading commit data...
ffmpeg.h Loading commit data...
ffmpeg_cuvid.c Loading commit data...
ffmpeg_dxva2.c Loading commit data...
ffmpeg_filter.c Loading commit data...
ffmpeg_opt.c Loading commit data...
ffmpeg_qsv.c Loading commit data...
ffmpeg_vaapi.c Loading commit data...
ffmpeg_vdpau.c Loading commit data...
ffmpeg_videotoolbox.c Loading commit data...
ffplay.c Loading commit data...
ffprobe.c Loading commit data...
ffserver.c Loading commit data...
ffserver_config.c Loading commit data...
ffserver_config.h Loading commit data...
library.mak Loading commit data...
version.sh Loading commit data...