Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Recently we encountered an issue when decoding a h.263 file:

FFmpeg will freeze when decoding h.263 video with NVDEC. Turns out this is not directly related to NVDEC but is a problem that shows with several other HW decoders like VDPAU, though the exact kind of error is different (either error messages or freezing[1]). The root cause is that ff_thread_finish_setup() is called twice per frame from ff_h263_decode_frame(). This is not supported by ff_thread_finish_setup() and specifically checked for and warned against in the functions code. The issue is also specific to hw accelerated decoding only as the second call to ff_thread_finish_setup() is only issued when hw acceleration is on. The fix is simple: add a check that the first call is only send when hw acceleration is off, and the second call only when hw acceleration is on (see attached patch). This works fine as far as I was able to test with vdpau and nvdec/nvcuvid hw decoding. The patch also adds NVDEC to the hw config list if available.

I also noticed a secondary issue when browsing through the code which is that, according to documentation, ff_thread_finish_setup() should only be called if the codec implements update_thread_context(), which h263dec does not. The patch does not address this and I'm not sure any action needs to be taken here at all.

[1] This is depending on whether or not the hw decoder sets the  HWACCEL_CAPS_ASYNC_SAFE flag
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

avcodec/h263dec: Fixed VA API, VDPAU, and VideoToolbox hardware acceleration due to missing `hw_configs` property.

Bug originally introduced in commit 758fbc54.

Since db772308 parsing of
mpeg4-extradata lead to a "Failed to parse extradata" warning, because
ff_mpeg4_decode_picture_header returns AVERROR_INVALIDDATA in case that
no VOP was found. This patch adds a parameter to signify whether a
header (where the absence of a VOP does not raise an error) or not is
parsed. The first mode is of course used for parsing headers.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The profile field is changed by code inside and outside the decoder,
its not a reliable indicator of the internal codec state.
Maintaining it consistency with studio_profile is messy.
Its easier to just avoid it and use only studio_profile

Fixes: assertion failure
Fixes: ffmpeg_crash_9.avi

Found-by: Thuan Pham, Marcel Böhme, Andrew Santosa and Alexandru Razvan Caciulescu with AFLSmart
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The profile after reading headers can be different from when the context was initialized
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes infinite loop
Fixes: 6858/clusterfuzz-testcase-ffmpeg_AV_CODEC_ID_MSMPEG4V3_fuzzer-4681563766784000
Fixes: 6890/clusterfuzz-testcase-ffmpeg_AV_CODEC_ID_WMV1_fuzzer-4756103142309888

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

This is a profile supporting > 8-bit video and has a higher quality DCT

This includes a pointer to the associated hwaccel for decoders using
hwaccels - these will be used later to implement the hwaccel setup
without needing a global list.

Also added is a new file listing all hwaccels as external declarations -
this will be used later to generate the hwaccel list at configure time.

This includes a pointer to the associated hwaccel for decoders using
hwaccels - these will be used later to implement the hwaccel setup
without needing a global list.

Also added is a new file listing all hwaccels as external declarations -
this will be used later to generate the hwaccel list at configure time.

This was predictably nightmarish, given how ridiculous mpeg4 is.
I had to stare at the cuvid parser output for a long time to work
out what each field was supposed to be, and even then, I still don't
fully understand some of them. Particularly:

vop_coded: If I'm reading the decoder correctly, this flag will always
           be 1 as the decoder will not pass the hwaccel any frame
           where it is not 1.
divx_flags: There's obviously no documentation on what the possible
            flags are. I simply observed that this is '0' for a
            normal bitstream and '5' for packed b-frames.
gmc_enabled: I had a number of guesses as to what this mapped to.
             I picked the condition I did based on when the cuvid
             parser was setting flag.

Also note that as with the vdpau hwaccel, the decoder needs to
consume the entire frame and not the slice.

The new name more accuratly describes what the function does
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Merges Libav commit 8dfba25c.
Signed-off-by: wm4 <nfxjfg@googlemail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Diego Biurrun <diego@biurrun.de>

Signed-off-by: Anton Khirnov <anton@khirnov.net>

Libav, for some reason, merged this as a public API function. This will
aid in future merges.

A define is left for backwards compat, just in case some person
used it, since it is in a public header.
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>

The VDPAU API never explicitly supported H.263 in the first place.
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Deprecate older VA pixel formats (MOCO, IDCT) as it is now very unlikely
to ever be useful in the future. Only keep plain AV_PIX_FMT_VAAPI format
that is aliased to the older VLD variant.

This is an API change.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>

Express bitfields more simply.
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Convert doxygen to multiline and express bitfields more simply.
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

It is necessary to avoid circular header dependencies.

h264.h and hevc.h are mutually exclusive due to defining some of the same
names. As such, we need to avoid forcing h264.h to be included if we want
hevc decode acceleration to be possible.

However, some of the pre-hwaccel helper functions need h264.h. To avoid
messy collisions, let's move the declaration of all those helpers to
a separate header which we will exclude for the hevc support (which will
be hwaccel-only).
Signed-off-by: Philip Langdale <philipl@overt.org>

It is just a duplicate of an AVCodecContext member so use it instead.

They are just duplicates of AVCodecContext members so use those instead.

Fixes Ticket4542
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

If this causes problems then the stream_codec_tag should be enabled again
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>