The relative error between two encoding strategies is the simple
difference of rate-distortion values, and not the absolute
difference. An absolute measure would allow worsening of the
quantization error as well as improving.

1. Fix sf_idx and band_type addressing to address only the first
subwindow in the group (others could hold garbage values)

2. Don't step on ms_mask when is_mask is set. I/S selection
already sets the ms_mask properly and shouldn't be overridden.

3. Use mid/sid cb/sf when computing coding error, as should be
since those are the cb/sfs that will eventually be set.

4. Fix distortion computation on multi-subwindow groups (was
subtracting the bits terms multiple times)

5. Clear ms_mask when one side uses PNS and the other doesn't.
When using PNS, ms_mask signals correlated noise, which can be
detected just like regular M/S detection, so we don't skip
noise bands, but when only one side uses PNS setting the flag
can confuse some encoders, so avoid that.

Correct addressing of sf_idx and band_type arrays in I/S code.

Both arrays don't guarantee valid values for subwindows at all
times, depending on which coder is being used, which could result
in reading of garbage values. Instead, only the first subwindow
in the window group has to be used, ie: use w*16+g instead of
(w+w2)*16+g

Change the condition for application of the M/S transform to match
that of the decoder. Namely, that no special coding books must be
in use in either channel. While the condition ought to be
equivalent to the current one when the invariant of is_mask is
kept, matching the decoder's condition is safer and easier to
maintain.

In rare corner cases it could still fail an assert on sf_diff due
to failure to update prev_sf in some code paths. Fix that case.

Fixes an assertion error reported in #2686 that happens when
using prediction (either explicitly or implicitly by setting
the AAC main profile), since prediction code would allow
creating new zeroes or removing existing ones, without
properly checking for SF delta violations.

This patch forbids creating/removing zeroes, perhaps an
overly conservative approach, but a safe one. More permissive
and sophisticated approaches may be attempted in the future.

Fixes ticket #5151.

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes out of array read
Fixes: 03c406ec9530e594a074ce2979f8a1f0/asan_heap-oob_7dec26_4664_37c52495b2870a2eaac65f53958e76c1.flac

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

since:
cbcc88c0 libvpx: Support setting color range for vp9.
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: James Zern <jzern@google.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes integer overflow
Fixes: 0063df8be3aaa30dd6d76f59c8f818c8/signal_sigsegv_7b7b59_3634_bf418b6822bbfa68734411d96b667be3.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 0063df8be3aaa30dd6d76f59c8f818c8/signal_sigsegv_7b7b59_3634_bf418b6822bbfa68734411d96b667be3.mov
Fixes assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes memleaks
Fixes: 245dae71bc28fe8f3f454300277e575a/signal_sigabrt_7ffff6ae7cb7_5062_f7d9dfa385454806586076f9db48c0bf.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Use av_realloc() rather than av_malloc() when normalizing noncompliant
private data in get_qt_codec().
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Found-by: Leo Izen <leo.izen@gmail.com> as thebombzen on IRC
Signed-off-by: Timothy Gu <timothygu99@gmail.com>

Fixes assertion failure
Fixes out of memory access

Fixes: test_casex.ivf
Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Either disabling or init'ing secure memory is required after the use
of gcry_check_version. From a look at the functions rtmpdh uses, I
noticed none require the use of secure memory, so we disable it [1][2].

This resolves some errors returned by rtmpdh code with uninitialized
gcrypt, especifically:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

Version "1.5.4" was arbitrarily chosen. An older version probably works
as well, but I couldn't compile older versions to test on my machine.

[1]
https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
[2]
https://www.gnupg.org/documentation/manuals/gcrypt/Controlling-the-library.htmlSigned-off-by: Ricardo Constantino <wiiaboo@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

The transformation to bytes must happen after alignment to get the same
resulting pointers as before.

This fixes segmentation faults in the assembler code.

The regression was introduced in commit 95536898.
Reviewed-by: Kieran Kunhya <kierank@obe.tv>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

This exploits an approach based on the sieve of Eratosthenes, a popular
method for generating prime numbers.

Tables are identical to previous ones.

Tested with FATE with/without --enable-hardcoded-tables.

Sample benchmark (Haswell, GNU/Linux+gcc):
prev:
7860100 decicycles in cbrt_tableinit,       1 runs,      0 skips
7777490 decicycles in cbrt_tableinit,       2 runs,      0 skips
[...]
7582339 decicycles in cbrt_tableinit,     256 runs,      0 skips
7563556 decicycles in cbrt_tableinit,     512 runs,      0 skips

new:
2099480 decicycles in cbrt_tableinit,       1 runs,      0 skips
2044470 decicycles in cbrt_tableinit,       2 runs,      0 skips
[...]
1796544 decicycles in cbrt_tableinit,     256 runs,      0 skips
1791631 decicycles in cbrt_tableinit,     512 runs,      0 skips

Both small and large run count given as this is called once so small run
count may give a better picture, small numbers are fairly consistent,
and there is a consistent downward trend from small to large runs,
at which point it stabilizes to a new value.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

It is used to store the difference between pointers, so ptrdiff_t is the
correct type.

This prevents potential overflows.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

This fixes segmentation faults due to out of bounds writes, when
color_start is interpreted as negative number.

This regression was introduced in commit 57631f.
Reviewed-by: Mats Peterson <matsp888@yahoo.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

They are stored in a byte, thus more than 255 is not possible
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 1b8b83a53bfa751f01b1daa65a4758db/signal_sigabrt_7ffff6ae7cb7_7488_403f71d1a2565b598d01b6cb110fac8f.aiff
Fixes: assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 07fa81f300736485e585b63e7b1cd155/signal_sigabrt_7ffff6ae7cb7_16_94ee815791a1357826524072db5bc2dc.dfa

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This patch adds a new static function get_qt_codec() that takes care of
the initial retrieval of the fourcc and codec ID for A_QUICKTIME and
V_QUICKTIME. It also normalizes noncompliant private data found in some
older files that incorrectly starts with the fourcc by expanding/shifting
the data by 4 bytes, and storing the data size at the start. This is
necessary in order for the rest of the code in the A_QUICKTIME and
V_QUICKTIME blocks (and most likely other code as well) to correctly
parse the private data.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes index out of bounds error
Fixes: aac_index_out_of_bounds.wmv
Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: dvdec_left_shift.avi
Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes invalid shift
Fixes: g2m_left_shift_2.wmv
Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Use monospaced font, and a black box outline.

Fixes: mjpeg_left_shift.avi
Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes ticket #5142.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Marton Balint <cus@passwd.hu>