Split away option settings, sanity checks and general setup.

Mostly cosmetic changes adding some intermediate to shorten the lines.

This takes into account planar vs. packed when comparing to find the closest
sample format match.

Signed-off-by: Diego Biurrun <diego@biurrun.de>

If there is an error during frame parsing, but AVCodecContext.channels was
changed and AC3DecodeContext.out_channels was set previously, the two may not
match.

Fixes CVE-2012-2802
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

CC: libav-stable@libav.org

When decode_nal_units() previously encountered a NAL_END_SEQUENCE,
and there are some junk bytes left in the input buffer, but no start codes,
buf_index gets stuck 3 bytes before the end of the buffer.

This can trigger an infinite loop in the caller code, eg. in
try_decode_trame(), as avcodec_decode_video() then keeps returning zeroes,
with 3 bytes of the input packet still available.

With this change, the remaining bytes are skipped so the whole packet gets
consumed.

CC:libav-stable@libav.org
Signed-off-by: Jindřich Makovička <makovick@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>

It is got_output, not data_size.

Avoids a forward declaration of decode_chunks().

The decode function assumes that the video will have those dimensions.

Fixes CVE-2012-2801

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Fixes CVE-2012-2799

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Fixes CVE-2012-2798

CC:libav-stable@libav.org

Fixes a part of CVE-2012-2795

CC:libav-stable@libav.org

Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

Fixes a part of CVE-2012-2795

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Fixes a part of CVE-2012-2795

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>

This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2012-2800

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

It's not used outside of ivi_common.c

This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Fixes CVE-2012-2794

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Fixes CVE-2012-2776.

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

Fixes CVE-2012-2792

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind