Fixes: Timeout (89sec -> 7sec)
Fixes: 17035/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_BINK_fuzzer-5737222422134784

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegReviewed-by: Peter Ross <pross@xvid.org>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: Timeout (195sec -> 2ms)
Fixes: 16735/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5090676403863552

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #8154.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

When the mov/mp4 demuxer encounters an error during decrypting a packet,
it returns the error, yet doesn't free the packet, so that the packet
leaks. This has been fixed in this commit.

Fixes the memleaks from ticket #8150.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 15295/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HEVC_fuzzer-5675655187922944

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

ttaenc contained (1 << unary) - 1 as an argument for a function
expecting an unsigned int. unary can be as big as 31 in this case.
The type of the shift and the whole expression is int, because 1 fits
into an integer, so that the behaviour is undefined if unary == 31
as the result of the shift can't be represented in an int §. Subtraction
by 1 (which makes the result of the whole expression representable in
an int) doesn't change that this is undefined (it usually leads to
signed integer overflow which is undefined, too).

The solution is simple: Make 1 unsigned to change the type of the
whole expression to unsigned int (as the function expects anyway).

Fixes ticket #8153.

§: This of course presupposes the common int range of -2^31..2^31-1
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #4799.
Analyzed-by: Aleksandr Ustinov

It's considerably faster than libaom in most systems.

Reviewed-by: BBB
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Rick Kern <kernrj@gmail.com>

Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: Rick Kern <kernrj@gmail.com>

On iOS 11, encoding a frame may return error with log
"Error encoding frame 0", which means vtenc_output_callback
is called with status=0 and sample_buffer=NULL. Then the
encoding session will be crashed on next callback wether or not
closing the codec context.

Let us look through the link below introducing VTCompressionOutputCallback,

https://developer.apple.com/documentation/videotoolbox/vtcompressionoutputcallback?language=objc

"status=0" (noErr) means compression was successful.
"sampleBuffer=NULL" means the frame was dropped when compression
was successful (status=0) or compression was not successful (status!=0).

So we should not set AVERROR_EXTERNAL on "status=0" and "sample_buffer=NULL"
as it is not a error.

The fix is that we only set AVERROR_EXTERNAL with status value non zero.
When sample_buffer is NULL and status value is zero, we simply return
with no other operation.

This crash often occurs on iOS 11 for example encoding 720p@25fps.
Signed-off-by: sharpbai <sharpbai@gmail.com>
Signed-off-by: Rick Kern <kernrj@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 65535 * 65312 cannot be represented in type 'int'
Fixes: 16704/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-6294115603447808

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 57342 * 120830 cannot be represented in type 'int'
Fixes: 16426/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_CFHD_fuzzer-5758744817827840

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg

snow allows creating large output from tiny input, like other
wavelet codecs

Fixes: Timeout (82sec -> 1.5sec)
Fixes: 9520/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_SNOW_fuzzer-6286612576075776

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This avoids 3 warnings
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: Multiple memleaks
Fixes: ffmpeg-memory-leak
Found-by: Francis Provencher <francis@protekresearchlab.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: Timeout (151sec -> 173ms)
Fixes: 16053/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MOTIONPIXELS_fuzzer-5647069169057792
Fixes: 16053/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MOTIONPIXELS_fuzzer-5108957126852608

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>