When performing a comparison of a signed int and an unsigned int, the
signed int is first converted to an unsigned int, so that negative
values are being treated as big, positive values. This can become a
problem in an overread check, namely when an overread already happened.
So change the type of the variable containing the amount of bits that
need to be left to signed.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

The maximum allowed index for an array access is FF_ARRAY_ELEMS - 1; yet
the current code allowed FF_ARRAY_ELEMS. This wasn't dangerous in practice,
as parameter sets with invalid ids were already filtered out during
reading.

Found via PVS-Studio (see ticket #8156).
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

header.data_max and header.data_min are not necessarely set on all decoding scenarios.

Fixes a Valgrind reported regression since cfa19377.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

Up until now, read_frame_internal always initialized the packet it
received. But since the recent changes to ff_read_packet, this is no
longer needed: If the parsing queue is initially empty upon entering
read_frame_internal, the packet will now either contain content upon
success or be blank upon failure of ff_read_packet. If the parsing
queue is initially not empty, the packet will be overwritten with the
oldest one from the parsing queue.

Similarly, it is unnecessary to initialize ret in read_frame_internal.

In parse_packet, it is easily possible to only initialize the packet
used as temporary storage for the output if said packet is used at all;
furthermore, this packet doesn't need to be zero-initialized, because
av_init_packet will initialize every field except size and data and
those fields will be set by av_parser_parse2.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Up until now, parse_packet() used a stack packet in case the stream is
flushed. But using such a packet is unnecessary as there is an AVPacket
readily available, it just needs to be used. Whether flushing is intended
or not will now be signalled by an explicit parameter rather than by
whether the packet parameter is NULL. This removes a few checks in
parse_packet(), gets rid of the initialization of the stack packet and
also reduces usage of sizeof(AVPacket) in libavformat.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Up until now, read_frame_internal in avformat/utils.c uses a spare
packet on the stack that serves no real purpose: At no point in this
function is there a need for another packet besides the packet destined
for output:
1. If the packet doesn't need a parser, but is output as is, the content
of the spare packet (that at this point contains a freshly read packet)
is simply copied into the output packet (via simple assignment, not
av_packet_move_ref, thereby confusing ownership).
2. If the packet needs parsing, the spare packet will be reset after
parsing and any packets resulting from the packet read will be put into
a packet list; the output packet is not used here at all.
3. If the stream should be discarded, the spare packet will be
unreferenced; the output packet is not used here at all either.

Therefore the spare packet and the copies can be removed in principle.
In practice, one more thing needs to be taken care of: If ff_read_packet
failed, the output packet was not affected, now it is. But given that
ff_read_packet returns a blank (as if reset via av_packet_unref) packet
on failure, there is no problem from this side either.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

When AVFMT_FLAG_GENPTS is set, av_read_frame would put a reference to a
packet in the packet list (via av_packet_ref) and then immediately
thereafter unreference the original packet. This has been changed to
move the reference instead.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Up until now, avformat_find_stream_info had a potential for memleaks:
When everything was fine, it read packets and (depending upon whether
AVFMT_FLAG_NOBUFFER was set) put them in a packet list or unreferenced
them when they were no longer needed. But upon failure, said packets
would leak if they were not already on the packet list. This patch fixes
this.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Since the recent changes to ff_packet_list_put, the source packet will
be automatically reset when the reference is moved to the packet list,
so that it is unnecessary to reinitialize the packet in the loops in
parse_packet and ff_read_packet; initializing once at the beginning is
enough.

This also fixes a potential, but currently unexisting problem: If the
raw packet buffer was initially not empty and probe_codec() failed,
then the packet returned would not be initialized. But given that
probe_codec() currently can't fail (always returns 0) this was not an
acute danger.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

This is shorter and consistent across filters.

Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: Marton Balint <cus@passwd.hu>

Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: Marton Balint <cus@passwd.hu>

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
Signed-off-by: Marton Balint <cus@passwd.hu>

Fixes: Timeout (30sec -> 25sec)
Fixes: 17050/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MOTIONPIXELS_fuzzer-5719149803732992

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/ituh263dec: Make the condition for the studio slice start code match between ff_h263_resync() and ff_mpeg4_decode_studio_slice_header()

If they mismatch an infinite loop can occur
Fixes: Timeout (infinite loop)
Fixes: 17043/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG4_fuzzer-5695051748868096

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: -1094995519 * 64 cannot be represented in type 'int'
Fixes: 17030/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RALF_fuzzer-5640695838146560

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: left shift of 1073741824 by 1 places cannot be represented in type 'int'
Fixes: 16976/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1_fuzzer-4847262047404032

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: left shift of negative value -2
Fixes: 16964/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5757853565976576

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: left shift of negative value -7
Fixes: 16959/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3_fuzzer-5200360825683968

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: overflow in aspect ratio calculation
Fixes: signed integer overflow: 393215 * 14594 cannot be represented in type 'int'
Fixes: 15728/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3IMAGE_fuzzer-5661588893204480

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

"9.1.1.43 P Reference Distance (REFDIST)"
"The value of REFDIST shall be less than, or equal to, 16."
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

We need to reset samples in this case to avoid being stuck with incorrect
samples value.

Fixes: OOM
Fixes: Timeout
Fixes: 16627/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5638059583864832
Fixes: 17089/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5672188463546368

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: negation of -2147483648 cannot be represented in type 'int'; cast to an unsigned type to negate this value to itself
Fixes: signed integer overflow: -14527961 - 2147483425 cannot be represented in type 'int'
Fixes: 16380/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5645957131141120
Fixes: 16968/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5716169901735936
Fixes: 17074/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5198710497083392

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Values larger would fail subsequent tests.

Fixes: signed integer overflow: 5 + 2147483646 cannot be represented in type 'int'
Fixes: 16966/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_HEVC_fuzzer-5695709549953024

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Affected every usage of vf_xbr, e.g. the FATE-tests filter-2xbr,
filter-3xbr, filter-4xbr.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Affected every usage of this filter; in particular, it affected the
FATE-tests filter-2xbr, filter-3xbr and filter-4xbr.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Affected the vsynth*-jpeg2000 and the vsynth*-jpeg2000-97 FATE tests
(where * ranges over { 1, 2, 3, _lena }) as well as ticket #7983.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #8160.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Affected 61 FATE-tests: 60 vsynth tests and lavf-mxf_opatom.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Affected the FATE-tests vsynth_lena-dv-411, vsynth1-dv-411,
vsynth2-dv-411 and hevc-paramchange-yuv420p.yuv420p10.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavcodec/ffv1enc.c accessed an array of uint8_t [32] via array[0][j]
in order to loop over all the uint8_t in this array of arrays. Of course
this implied an out-of-bounds access for array[0] and UBSan complained
about this. So replace this with nested loops; furthermore, factor this
out into a function of its own to easily break out of the nested loops.

This affected the FATE-tests vsynth1-ffv1, vsynth1-ffv1-v3-yuv420p,
vsynth1-ffv1-v3-yuv422p10, vsynth1-ffv1-v3-yuv444p16,
vsynth1-ffv1-v3-bgr0, vsynth1-ffv1-ffv1-v3-rgb48 as well as the
corresponding vsynth2-*, vsynth3-* and the vsynth_lena-* tests.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This affected many FATE-tests: The number of failing tests went down
from 663 to 344. (Both numbers exclude tests that failed because of
unaligned accesses in code that is inside #if HAVE_FAST_UNALIGNED.)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Lou Logan <lou@lrcd.com>

Signed-off-by: Lou Logan <lou@lrcd.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>