Fixes: timeout in 730/clusterfuzz-testcase-5265113739165696 (part 1 of 2)

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Reviewed-by: BBB
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Update the fate test as needed.

Update the fate test as needed.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

use ff_thread_once
Suggested-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Muhammad Faiz <mfcc64@gmail.com>

use ff_thread_once
Suggested-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Muhammad Faiz <mfcc64@gmail.com>

use ff_thread_once
Suggested-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Muhammad Faiz <mfcc64@gmail.com>

use ff_thread_once
Suggested-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Muhammad Faiz <mfcc64@gmail.com>

The constants used in the decoder used floating point precision,
and this caused different values to be generated on different
architectures.

So, eradicate floating point numbers and use fixed point (32.32)
arithmetics everywhere, replacing constants with precomputed integer
values.

Signed-off-by: Vittorio Giovara <vittorio.giovara at gmail.com>
Signed-off-by: Paul B Mahol <onemda@gmail.com>

The way videotoolbox hooks in as a hwaccel is pretty hacky. The VT decode
API is not invoked until end_frame(), so alloc_frame() returns a dummy
frame with a 1-byte buffer. When end_frame() is eventually called, the
dummy buffer is replaced with the actual decoded data from
VTDecompressionSessionDecodeFrame().

When the VT decoder fails, the frame returned to the h264 decoder from
alloc_frame() remains invalid and should not be used. Before
97472199, it was accidentally being
returned all the way up to the API user. After that commit, the dummy
frame was unref'd so the user received an error.

However, since that commit, VT hwaccel failures started causing random
segfaults in the h264 decoder. This happened more often on iOS where the
VT implementation is more likely to throw errors on bitstream anomolies.
A recent report of this issue can be see in
http://ffmpeg.org/pipermail/libav-user/2016-November/009831.html

The issue here is that the dummy frame is still referenced internally by the
h264 decoder, as part of the reflist and cur_pic_ptr. Deallocating the
frame causes assertions like this one to trip later on during decoding:

  Assertion h->cur_pic_ptr->f->buf[0] failed at src/libavcodec/h264_slice.c:1340

With this commit, we leave the dummy 1-byte frame intact, but avoid returning it
to the user.

This reverts commit 97472199.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Signed-off-by: James Zern <jzern@google.com>

The code previously completely discarded frames that had any error in a slice
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Without this the FPU state becomes trashed and causes mysterious
fate failures with cpuflags=0
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes memleak
Fixes: 548/clusterfuzz-testcase-5511470875934720

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Ever since the codecpar changes, this has been always printed when
opening a flv file. This is because the codecpar changes made all
streams to be added lazily as read_packet is called.

Now the previous commit takes care of this instead.

There is no reason that draining couldn't return an error or two. But
some decoders don't handle this very well, and might always return an
error. This can lead to API users getting into an infinite loop and
burning CPU, because no progress is made and EOF is never returned.

In fact, ffmpeg.c contains a hack against such a case. It is made
unnecessary with this commit, and removed with the next one. (This
particular error case seems to have been fixed since the hack was
added, though.)

This might lose frames if decoding returns errors during draining.

kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey is not available in
iOS

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes: 723/clusterfuzz-testcase-6471394663596032

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/dcadsp: Fix 2 runtime error: signed integer overflow: -1958094138 - 1078906344 cannot be represented in type 'int'

Fixes: 722/clusterfuzz-testcase-5711268868521984

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: undefined behavior in 717/clusterfuzz-testcase-5434924129583104

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/mpeg4videodec: Fix runtime error: signed integer overflow: 134527392 * 16 cannot be represented in type 'int'

This checks the sprite delta intermediates for overflow
Fixes: 716/clusterfuzz-testcase-4890287480504320

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 709/clusterfuzz-testcase-4789836449841152

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 710/clusterfuzz-testcase-5091051431788544

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 705/clusterfuzz-testcase-5129572590813184

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Also clear the state on errors

Fixes integer overflows in 701/clusterfuzz-testcase-6594719951880192

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes timeout with 700/clusterfuzz-testcase-5660909504561152
Fixes timeout with 702/clusterfuzz-testcase-4553541576294400

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

fix ticket: #6204
Signed-off-by: Steven Liu <lq@chinaffmpeg.org>

Found-by: Miles
Signed-off-by: James Almer <jamrial@gmail.com>

The code modifying the buffer on big endian systems was removed.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

This makes ubsan happy and also considerably increases performance on
big endian systems.

Tested on an IBM POWER7 3.55 GHz

Before:

2.24user 0.14system 0:02.39elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
2.26user 0.11system 0:02.38elapsed 99%CPU (0avgtext+0avgdata 2688maxresident)k
2.23user 0.15system 0:02.38elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
2.25user 0.12system 0:02.38elapsed 100%CPU (0avgtext+0avgdata 2624maxresident)k
2.20user 0.15system 0:02.36elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k

After:

1.86user 0.13system 0:02.00elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.89user 0.11system 0:02.01elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.85user 0.14system 0:02.00elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.84user 0.15system 0:01.99elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.89user 0.13system 0:02.02elapsed 99%CPU (0avgtext+0avgdata 2688maxresident)k
Tested-by: Nicolas George <george@nsup.org>
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This adds "-profile[:v] profile_name"-style option.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

The filter field is often used to check whether a filter is
configured. If configuring the filter actually fails somewhere in
the middle of it, these fields could still be set to non-NULL, which
lead to other code accessing the half-configured filter graph, which
in turn could lead to crashes within libavfilter.

Solve this by properly resetting all fields.

This was triggered by a fuzzed sample after the recent changes. It's
unknown whether this behavior could be triggered before that.

If a subtitle packet came before the first video frame could be fully
decoded, the subtitle packet would get discarded. This puts the subtitle
into a queue instead, and processes it once the attached filter graph is
initialized.