There is no reason that draining couldn't return an error or two. But
some decoders don't handle this very well, and might always return an
error. This can lead to API users getting into an infinite loop and
burning CPU, because no progress is made and EOF is never returned.

In fact, ffmpeg.c contains a hack against such a case. It is made
unnecessary with this commit, and removed with the next one. (This
particular error case seems to have been fixed since the hack was
added, though.)

This might lose frames if decoding returns errors during draining.

kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey is not available in
iOS

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes: 723/clusterfuzz-testcase-6471394663596032

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/dcadsp: Fix 2 runtime error: signed integer overflow: -1958094138 - 1078906344 cannot be represented in type 'int'

Fixes: 722/clusterfuzz-testcase-5711268868521984

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: undefined behavior in 717/clusterfuzz-testcase-5434924129583104

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/mpeg4videodec: Fix runtime error: signed integer overflow: 134527392 * 16 cannot be represented in type 'int'

This checks the sprite delta intermediates for overflow
Fixes: 716/clusterfuzz-testcase-4890287480504320

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 709/clusterfuzz-testcase-4789836449841152

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 710/clusterfuzz-testcase-5091051431788544

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 705/clusterfuzz-testcase-5129572590813184

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Also clear the state on errors

Fixes integer overflows in 701/clusterfuzz-testcase-6594719951880192

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes timeout with 700/clusterfuzz-testcase-5660909504561152
Fixes timeout with 702/clusterfuzz-testcase-4553541576294400

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

fix ticket: #6204
Signed-off-by: Steven Liu <lq@chinaffmpeg.org>

Found-by: Miles
Signed-off-by: James Almer <jamrial@gmail.com>

The code modifying the buffer on big endian systems was removed.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

This makes ubsan happy and also considerably increases performance on
big endian systems.

Tested on an IBM POWER7 3.55 GHz

Before:

2.24user 0.14system 0:02.39elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
2.26user 0.11system 0:02.38elapsed 99%CPU (0avgtext+0avgdata 2688maxresident)k
2.23user 0.15system 0:02.38elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
2.25user 0.12system 0:02.38elapsed 100%CPU (0avgtext+0avgdata 2624maxresident)k
2.20user 0.15system 0:02.36elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k

After:

1.86user 0.13system 0:02.00elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.89user 0.11system 0:02.01elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.85user 0.14system 0:02.00elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.84user 0.15system 0:01.99elapsed 99%CPU (0avgtext+0avgdata 2624maxresident)k
1.89user 0.13system 0:02.02elapsed 99%CPU (0avgtext+0avgdata 2688maxresident)k
Tested-by: Nicolas George <george@nsup.org>
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This adds "-profile[:v] profile_name"-style option.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

The filter field is often used to check whether a filter is
configured. If configuring the filter actually fails somewhere in
the middle of it, these fields could still be set to non-NULL, which
lead to other code accessing the half-configured filter graph, which
in turn could lead to crashes within libavfilter.

Solve this by properly resetting all fields.

This was triggered by a fuzzed sample after the recent changes. It's
unknown whether this behavior could be triggered before that.

If a subtitle packet came before the first video frame could be fully
decoded, the subtitle packet would get discarded. This puts the subtitle
into a queue instead, and processes it once the attached filter graph is
initialized.

Be more careful when an input stream encounters EOF when its filtergraph
has not been configured yet. The current code would immediately mark the
corresponding output streams as finished, while there may still be
buffered frames waiting for frames to appear on other filtergraph
inputs.

This should fix the random FATE failures for complex filtergraph tests
after a3a0230a

This merges Libav commit 94ebf556. It was previously skipped.

This is the last filter init related Libav commit that was skipped, so
this also removes the commits from doc/libav-merge.txt.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

This is a more appropriate place for it, and will also be useful in the
following commit.

This merges Libav commit d2e56cf7. It was previously skipped.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Broken by the previous Libav commit (even in Libav, thus a separate
commit).
Signed-off-by: wm4 <nfxjfg@googlemail.com>

This makes sure the actual stream parameters are used, which is
important mainly for hardware decoding+filtering cases, which would
previously require various weird workarounds to handle the fact that a
fake software graph has to be constructed, but never used.
This should also improve behaviour in rare cases where
avformat_find_stream_info() does not provide accurate information.

This merges Libav commit a3a0230a. It was previously skipped.

The code in flush_encoders() which sets up a "fake" format wasn't in
Libav. I'm not sure if it's a good idea, but it tends to give
behavior closer to the old one in certain corner cases.

The vp8-size-change gives different result, because now the size of
the first frame is used. libavformat reported the size of the largest
frame for some reason.

The exr tests now use the sample aspect ratio of the first frame. For
some reason libavformat determines 0/1 as aspect ratio, while the
decoder returns the correct one.

The ffm and mxf tests change the field_order values. I'm assuming
another libavformat/decoding mismatch.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

This will be useful in the following commit, after which the muxer
timebase is not always available when encoding.

This merges Libav commit 3e265ca5. It was previously skipped.

There are some changes with how/when the mux_timebase field is set,
because the Libav approach often causes a too imprecise time base
to be set. This is hard, because the muxer's write_header function
can readjust the timebase, at which point we might already have
encoded packets buffered. (It might be better to buffer them after
the encoder, instead of after all the timestamp handling logic
before muxing.)

The two FATE tests change because the output time base is raised
for subtitles. (Needed to avoid certain rounding issues in other
cases.)

Includes a minor merge fix by Mark Thompson, and

    avconv: Move rescale to stream timebase before monotonisation

also by Mark Thompson <sw@jkqxz.net>.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Some callers (like do_subtitle_out()) call this with an AVPacket that is
not refcounted. This can cause undefined behavior.

Calling av_packet_move_ref() does not make a packet refcounted if it
isn't yet. (And it can't be made to, because it always succeeds,
and can't return ENOMEM.)

Call av_packet_ref() instead to make sure it's refcounted.

I couldn't find a case that is fixed by this with the current code. But
it will fix the fate-pva-demux test with the later patches applied.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Fixes ticket #6206.

Found a case where we use size==0, the other related commits
remain needed, and should be sufficient to fix the original issue

This reverts commit 7e4f32f4.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 697/clusterfuzz-testcase-6111250582863872

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

avcodec/mpegaudiodec_template: Fix runtime error: signed integer overflow: 2053224902 + 2053224902 cannot be represented in type 'int'

Fixes: 696/clusterfuzz-testcase-5853632270434304

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 693/clusterfuzz-testcase-6109776066904064

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Based on 260de8a2 by Paul B Mahol.
Fixes ticket #6210.

Only I-frames are decoded for now.
Signed-off-by: Paul B Mahol <onemda@gmail.com>

Cherry-picked from Libav d30719e6.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Makes sure the output can be mapped as OpenGL texture.
This is what at least video players normally want.

If AVVideotoolboxContext.cv_pix_fmt_type is set to 0, don't set the
kCVPixelBufferPixelFormatTypeKey value on the VT decoder.

This makes VT output its native format, which can be much faster on
some hardware iterations (if the native format does not match with
the requested format, it will be converted, which is slow).

The default is still forcing nv12.