Fixes: signed integer overflow: 1024 + 2147483640 cannot be represented in type 'int'
Fixes: 4671/clusterfuzz-testcase-minimized-6027464343027712

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegReviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The existing logic overrides container metadata even in cases where the
container metadata must be trusted (e.g. HDR). The original spec had no
provision for specifying color volume, so many files rely on the
assumption of Rec. 709.

An update to the spec included a 'clv' field for explicitly signaling
that the container should be trusted in an existing bitfield in the
frame header, but the default of 0 from old encoders forces Rec. 709,
which would break any HDR stream. Because there is no place in DNxHR for
specifying a transfer function, DNxHR HDR files must include
container-level color information.

This patch maintains the existing behavior of choosing the 709 over the
601 matrix when container-level information is missing, and allows
container-level information to win if present.
Signed-off-by: Steven Robertson <steven@strobe.cc>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: out of array access
Fixes: poc.dnxhd

Found-by: Bingchang, Liu@VARAS of IIE
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

FF_ARRAY_ELEMS returns a size_t value.

Reviewed-by: ubitux
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Otherwise all thread's private contexts have the avctx pointer set to
the AVCodecContext of the first thread, which means all writes to
ctx->avctx->* (in e.g. read_header) are effectively race conditions.

Fixes fate-dnxhd under tsan.

Deprecated in 11/2013.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

10- and 12-bit DNxHR use the same DC coefficient decoding process and
VLC table, just with a different shift value. From SMPTE 2019-1:2016,
8.2.4 DC Coefficient Decoding:

"For 8-bit video sampling, the maximum value of η=11 and for
10-/12-bit video sampling, the maximum value of η=13."

A sample file will be uploaded to show that with this patch, things
decode correctly:
dnxhr_hqx_12bit_1080p_smpte_colorbars_davinci_resolve.mov
Signed-off-by: Steven Robertson <steven@strobe.cc>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

It's a small and simple function that can be inlined.

This removes one private symbol and should reduce object dependencies with the next
major bump
Signed-off-by: James Almer <jamrial@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 2abd25478c62a675f335fac00b467023/asan_static-oob_10aff98_1227_8811480c6ef1e970a7977ceb7e5a9958.mxf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The DC VLC table used is too small, fixing this requires a sample,
thus request a sample.
Some samples are said to work even though the table has the wrong size, thus
this is left enabled if the user enables experimental features.

Fixes: 2abd25478c62a675f335fac00b467023/asan_static-oob_10aff98_1227_8811480c6ef1e970a7977ceb7e5a9958.mxf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Approved-by: kurosu
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes null pointer dereference
Fixes: 5c9d1a6f74a12763fc7c9dd7834022b9/signal_sigsegv_11f78d9_1461_ecee3c5e7205457498e79b3ffaf21d0c.mxf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This allows more efficient access to the array as the level and flags
are contiguous. Around 4% faster coefficient decoding.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Profiles 1256 & 1270 (currently) signal at the frame header and MB
levels the colorspace used, either RGB or YUV. While a MB-level
varying colorspace is not supported, whether it is constant can be
tracked so as to determine the exact colorspace.

This requires having bitdepth and the ACT and 4:4:4 flags, in turn
needing the CID. Because setting those before having validated
enough things may result in invalid/unset DSP fucntions, setting
the bitdepth in the context is delayed.

It is not tested against a true RGB sequence, though.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

These are DNxHR profiles with the following properties:
- Variable size in a profile (property added in a previous commit),
  requiring variable-sized macroblock table;
- Variable bitdepth, up to 12 bits.
- Better validation of buffer sizes and positions
Signed-off-by: Christophe Gisquet <christophe.gisquet@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This a 4:4:4 10 bits profile, where image size is not fixed by the
profile, and which strays a bit outside the old frame header parsing
code.

Fixes ticket #4581 (DNxHR is not stricly supported, but that sequence is).
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Christophe Gisquet <christophe.gisquet@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Currently not used, but will be used to indicate that a CIDEntry field
is not set, because it is variable, and that checks should be adapted.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Rely more on the actual syntax from the specs (also seen in the
encoder code).
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Otherwise, the dsp may change without its permuation being applied.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The current one, while correct, does not yield the best possible
results. The specificiations suggest another formula, which results
in quality gains in the decoded output from fate tests. This
justifies changing said formula.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The coded size is a multiple of the macroblock size, which is 16.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

This bit is 1 in some samples, and seems to coincide with interlaced
mbs and CID1260. 2008 specs do not know about it, and maintain qscale
is 11 bits. This looks oversized, but may help larger bitdepths.

Currently, it leads to an obviously incorrect qscale value, meaning
its syntax is shifted by 1. However, reading 11 bits also leads to
obviously incorrect decoding: qscale seems to be 10 bits.

However, as most profiles still have 11bits qscale, the feature is
restricted to the CID1260 profile (this flag is dependent on
a higher-level flag located in the header).

The encoder writes 12 bits of syntax, last and first bits always 0,
which is now somewhat inconsistent with the decoder, but ends up with
the same effect (progressive + reserved bit).
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Each line is padded by the format, and errors are now reported and
stop the decoding. Around 5% speedup.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This only occur when an overrun in coefficient decoding is
detected.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fewer arguments, less duplicated code.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Around 3x speedup with 4 threads. Maybe more mb lines should be
batched per thread, but that's good enough for a first try.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>