Fixes: OOM
Fixes: 3541/clusterfuzz-testcase-minimized-6469958596820992

Adds support for decoding codeblock data larger than 8kb
Reduces decoder memory consumption

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

When -format_code is not specified autodetection will happen.
Signed-off-by: Marton Balint <cus@passwd.hu>

This is done to enable input format autodetection in decklink_dec.
Signed-off-by: Marton Balint <cus@passwd.hu>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: James Almer <jamrial@gmail.com>

Remove the broadcast instructions as well now that they are wide
enough.
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

better func separator
and add comment for the restore rgb planes10 declaration

add return value check to supress the build warning message like
"warning: ignoring return value" when use attribute -Wunused-result.
Signed-off-by: Jun Zhao <jun.zhao@intel.com>
Reviewed-by: 刘歧 <lq@chinaffmpeg.org>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: Steven Liu <lingjiujianke@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Jacob Trimble <modmaker@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Without this there can be multiple memory leaks for unrecognized
ogg streams.
Signed-off-by: Dale Curtis <dalecurtis@chromium.org>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

decode_user_data() attempts to create an integer |build|
value with 8 bits of spacing for 3 components. However
each component is an int32_t, so shifting each component
is undefined for values outside of the 8 bit range.

This patch simply clamps input to 8-bits per component
and prints out a warning that the values were clamped.
Signed-off-by: Dale Curtis <dalecurtis@chromium.org>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

this will simplify libvpxenc/dec.c and ensure more stable versions of
the codecs are present.
Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

jpeg2000_ict_float_c: 2296.0
jpeg2000_ict_float_sse: 628.0
jpeg2000_ict_float_avx: 317.0
jpeg2000_ict_float_fma3: 262.0
Signed-off-by: James Almer <jamrial@gmail.com>

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: James Almer <jamrial@gmail.com>

It is unused.
Signed-off-by: Marton Balint <cus@passwd.hu>

Fixes: runtime error: left shift of negative value -255
Fixes: 4037/clusterfuzz-testcase-minimized-5290998163832832

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

Previously the codec kept an entire copy of the SPS, and restarted the VT decoder
session whenever it changed. This fixed decoding errors in [1], as
described in 9519983c. On further inspection, that sample features an SPS change
from High/4.0 to High/3.2 while moving from one scene to another.

Yesterday I received [2], which contains minor SPS changes where the
profile and level do not change. These occur frequently and are not associated with
scene changes. After 9519983c, the VT decoder session is recreated unnecessarily when
these are encountered causing visual glitches.

This commit simplifies the state kept in the VTContext to include just the first three
bytes of the SPS, containing the profile and level details. This is populated initially
when the VT decoder session is created, and used to detect changes and force a restart.

This means minor SPS changes are fed directly into the existing decoder, whereas
profile/level changes force the decoder session to be recreated with the new parameters.

After this commit, both samples [1] and [2] playback as expected.

[1] https://s3.amazonaws.com/tmm1/videotoolbox/spschange.ts
[2] https://s3.amazonaws.com/tmm1/videotoolbox/spschange2.tsSigned-off-by: Aman Gupta <aman@tmm1.net>

This reverts commit 218ce1f6.