av_mallocz + av_init_packet leads to the same result as av_mallocz +
av_packet_unref, but faster.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This has been forgotten in d5a3a20d.

Found via PVS-Studio (see ticket #8156).
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

1. Instead of relying on ff_packet_list_get to get the oldest element in
an AVPacketList, ff_read_packet used its own ad-hoc code. Said code
forgot to set the end of the list to NULL if the last element of the
list has been removed, thereby leaving the list in an inconsistent state.
2. Furthermore, if the list was not empty, the oldest element of the
list would always be copied into another packet structure before it was
known whether the oldest entry of the list would be removed. This makes
the ownership confusing and potentially copies unnecessarily.

Both of these issues have been fixed. ff_packet_list_get is used now.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The documentation of ff_packet_list_get currently didn't match the
actual usage:
1. It said that the destination packet is supposed to be initialized.
But this makes no sense given that it will be overwritten completely and
flacenc, mp3enc and ttaenc ignored this.
2. ff_packet_list_get returns an int, although it can't fail in case the
packet list is not empty (for which there is an assert). Again, several
callers didn't check for any return value.
In both cases, the documentation has been adapted to match actual usage.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #8175
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Moritz Barsnick <barsnick@gmx.net>

Fixes: Timeout (89sec -> 7sec)
Fixes: 17035/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_BINK_fuzzer-5737222422134784

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegReviewed-by: Peter Ross <pross@xvid.org>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: Timeout (195sec -> 2ms)
Fixes: 16735/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5090676403863552

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #8154.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

When the mov/mp4 demuxer encounters an error during decrypting a packet,
it returns the error, yet doesn't free the packet, so that the packet
leaks. This has been fixed in this commit.

Fixes the memleaks from ticket #8150.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 15295/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HEVC_fuzzer-5675655187922944

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

ttaenc contained (1 << unary) - 1 as an argument for a function
expecting an unsigned int. unary can be as big as 31 in this case.
The type of the shift and the whole expression is int, because 1 fits
into an integer, so that the behaviour is undefined if unary == 31
as the result of the shift can't be represented in an int §. Subtraction
by 1 (which makes the result of the whole expression representable in
an int) doesn't change that this is undefined (it usually leads to
signed integer overflow which is undefined, too).

The solution is simple: Make 1 unsigned to change the type of the
whole expression to unsigned int (as the function expects anyway).

Fixes ticket #8153.

§: This of course presupposes the common int range of -2^31..2^31-1
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #4799.
Analyzed-by: Aleksandr Ustinov

It's considerably faster than libaom in most systems.

Reviewed-by: BBB
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Rick Kern <kernrj@gmail.com>

Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: Rick Kern <kernrj@gmail.com>

On iOS 11, encoding a frame may return error with log
"Error encoding frame 0", which means vtenc_output_callback
is called with status=0 and sample_buffer=NULL. Then the
encoding session will be crashed on next callback wether or not
closing the codec context.

Let us look through the link below introducing VTCompressionOutputCallback,

https://developer.apple.com/documentation/videotoolbox/vtcompressionoutputcallback?language=objc

"status=0" (noErr) means compression was successful.
"sampleBuffer=NULL" means the frame was dropped when compression
was successful (status=0) or compression was not successful (status!=0).

So we should not set AVERROR_EXTERNAL on "status=0" and "sample_buffer=NULL"
as it is not a error.

The fix is that we only set AVERROR_EXTERNAL with status value non zero.
When sample_buffer is NULL and status value is zero, we simply return
with no other operation.

This crash often occurs on iOS 11 for example encoding 720p@25fps.
Signed-off-by: sharpbai <sharpbai@gmail.com>
Signed-off-by: Rick Kern <kernrj@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 65535 * 65312 cannot be represented in type 'int'
Fixes: 16704/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-6294115603447808

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 57342 * 120830 cannot be represented in type 'int'
Fixes: 16426/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_CFHD_fuzzer-5758744817827840

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg

snow allows creating large output from tiny input, like other
wavelet codecs

Fixes: Timeout (82sec -> 1.5sec)
Fixes: 9520/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_SNOW_fuzzer-6286612576075776

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>