• sharpbai's avatar
    avcodec/videotoolboxenc: fix encoding frame crash on iOS 11 · 6966548c
    sharpbai authored
    On iOS 11, encoding a frame may return error with log
    "Error encoding frame 0", which means vtenc_output_callback
    is called with status=0 and sample_buffer=NULL. Then the
    encoding session will be crashed on next callback wether or not
    closing the codec context.
    
    Let us look through the link below introducing VTCompressionOutputCallback,
    
    https://developer.apple.com/documentation/videotoolbox/vtcompressionoutputcallback?language=objc
    
    "status=0" (noErr) means compression was successful.
    "sampleBuffer=NULL" means the frame was dropped when compression
    was successful (status=0) or compression was not successful (status!=0).
    
    So we should not set AVERROR_EXTERNAL on "status=0" and "sample_buffer=NULL"
    as it is not a error.
    
    The fix is that we only set AVERROR_EXTERNAL with status value non zero.
    When sample_buffer is NULL and status value is zero, we simply return
    with no other operation.
    
    This crash often occurs on iOS 11 for example encoding 720p@25fps.
    Signed-off-by: 's avatarsharpbai <sharpbai@gmail.com>
    Signed-off-by: 's avatarRick Kern <kernrj@gmail.com>
    6966548c
videotoolboxenc.c 84.2 KB