• Aman Gupta's avatar
    avcodec/h264, videotoolbox: fix crash after VT decoder fails · b6eaa392
    Aman Gupta authored
    The way videotoolbox hooks in as a hwaccel is pretty hacky. The VT decode
    API is not invoked until end_frame(), so alloc_frame() returns a dummy
    frame with a 1-byte buffer. When end_frame() is eventually called, the
    dummy buffer is replaced with the actual decoded data from
    VTDecompressionSessionDecodeFrame().
    
    When the VT decoder fails, the frame returned to the h264 decoder from
    alloc_frame() remains invalid and should not be used. Before
    97472199, it was accidentally being
    returned all the way up to the API user. After that commit, the dummy
    frame was unref'd so the user received an error.
    
    However, since that commit, VT hwaccel failures started causing random
    segfaults in the h264 decoder. This happened more often on iOS where the
    VT implementation is more likely to throw errors on bitstream anomolies.
    A recent report of this issue can be see in
    http://ffmpeg.org/pipermail/libav-user/2016-November/009831.html
    
    The issue here is that the dummy frame is still referenced internally by the
    h264 decoder, as part of the reflist and cur_pic_ptr. Deallocating the
    frame causes assertions like this one to trip later on during decoding:
    
      Assertion h->cur_pic_ptr->f->buf[0] failed at src/libavcodec/h264_slice.c:1340
    
    With this commit, we leave the dummy 1-byte frame intact, but avoid returning it
    to the user.
    
    This reverts commit 97472199.
    Signed-off-by: 's avatarwm4 <nfxjfg@googlemail.com>
    b6eaa392
h264dec.c 37.3 KB