Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes: out of array reads
Fixes: 07e8b9c5d348ccdf7add0f37de20cf6c/asan_heap-oob_27e8df7_6849_e56653f768070ec8cb52f587048444c2.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This avoids running various table inits unnecessarily
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This makes SWS more robust
Fixes: 07650a772d98aa63b0fed6370dc89037/asan_heap-oob_27ddeaf_2657_2c81ff264dee5d9712cb3251fb9c3bbb.264
Fixes: out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This is quite an accurate approximation; testing shows ~ 2ulp error in
the floating point result. Tested with FATE.

Alternatively, if one wants "full accuracy", one can use powf, or sqrt
instead of sqrtf. With powf, one gets 1 ulp error (theoretically should be 0, as
0.75 is exactly representable) on GNU libm, with sqrt, 0 ulp error.
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

Abstracted into pos_pow34 utility function
Signed-off-by: Claudio Freire <klaussfreire@gmail.com>

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

src and dst are only 8-bit-aligned, so accessing them as uint16_t causes
SIGBUS crashes on architectures like sparc.

This fixes ubsan runtime error: load of misaligned address for type
'const uint16_t', which requires 2 byte alignment
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes out of array read
Fixes: 04442da73d935b776d2236282588d4f9/signal_sigsegv_2625a69_8790_ae85ffc889070663319b3417ede777b0.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes a warning with -Wformat-extra-args

decoding changes from 17 to 20 fps

Reviewed-by; Rostislav Pehlivanov <atomnuker@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This avoids closing and opening the bit reader
Reviewed-by: Rostislav Pehlivanov <atomnuker@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

All MDCT outputs must be checked in case of 128point MDCTs
Fixes: out of array read
Fixes: 04442da73d935b776d2236282588d4f9/signal_sigsegv_2625a69_351_52ca6226eb83547a2d26e322ce84ed84.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Use the ability to invert phase with ms_mask instead of changing
the codebook when possible, to avoid having to switch codebooks
if some bands are INTENSTY_BT and others are INTENSITY_BT2, since
usually a set ms_mask uses less bits that a codebook change. While
it may not always be a win (ie: if it causes an ms_mask bitmap
to be sent when it wouldn't have been otherwise), it's unlikely
since the ms_mask bitmap will almost always be there already for
M/S itself.

The relative error between two encoding strategies is the simple
difference of rate-distortion values, and not the absolute
difference. An absolute measure would allow worsening of the
quantization error as well as improving.

1. Fix sf_idx and band_type addressing to address only the first
subwindow in the group (others could hold garbage values)

2. Don't step on ms_mask when is_mask is set. I/S selection
already sets the ms_mask properly and shouldn't be overridden.

3. Use mid/sid cb/sf when computing coding error, as should be
since those are the cb/sfs that will eventually be set.

4. Fix distortion computation on multi-subwindow groups (was
subtracting the bits terms multiple times)

5. Clear ms_mask when one side uses PNS and the other doesn't.
When using PNS, ms_mask signals correlated noise, which can be
detected just like regular M/S detection, so we don't skip
noise bands, but when only one side uses PNS setting the flag
can confuse some encoders, so avoid that.

Correct addressing of sf_idx and band_type arrays in I/S code.

Both arrays don't guarantee valid values for subwindows at all
times, depending on which coder is being used, which could result
in reading of garbage values. Instead, only the first subwindow
in the window group has to be used, ie: use w*16+g instead of
(w+w2)*16+g

Change the condition for application of the M/S transform to match
that of the decoder. Namely, that no special coding books must be
in use in either channel. While the condition ought to be
equivalent to the current one when the invariant of is_mask is
kept, matching the decoder's condition is safer and easier to
maintain.

In rare corner cases it could still fail an assert on sf_diff due
to failure to update prev_sf in some code paths. Fix that case.

Fixes an assertion error reported in #2686 that happens when
using prediction (either explicitly or implicitly by setting
the AAC main profile), since prediction code would allow
creating new zeroes or removing existing ones, without
properly checking for SF delta violations.

This patch forbids creating/removing zeroes, perhaps an
overly conservative approach, but a safe one. More permissive
and sophisticated approaches may be attempted in the future.

Fixes ticket #5151.

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes out of array read
Fixes: 03c406ec9530e594a074ce2979f8a1f0/asan_heap-oob_7dec26_4664_37c52495b2870a2eaac65f53958e76c1.flac

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

since:
cbcc88c0 libvpx: Support setting color range for vp9.
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>