Also prevent a memory leak.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

The to_free AVframe must be freed just like the other ones.
Indeed, the calling application may expect all frames to be
released.

(This regression caused use-after-free in VLC with hwaccel.)
Signed-off-by: Rémi Denis-Courmont <remi@remlab.net>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

The code using the returned buffer might expect it to be
FF_INPUT_BUFFER_PADDING_SIZE padded as any other avpacket.

Prevent buffer overread.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Directly return from code 1, 2 and 6 codepaths and simplify the
remaining one to have a single overflow check and a single call to
mcdc.

Check for out of picture macroblocks before calling mcdc.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Make sure the value of wlog2 is always between 0 and 3.

Partially mitigate out of memory writes.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Split sound and video tag parsing in separate functions.

Signed-off-by: Martin Storsjö <martin@martin.st>

This also simplifies the code.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>

And optionally forward them to the caller instead of concealing them.

Unify err and ret in a single variable.

At least 2 bytes from the source are read every loop.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Return the proper error values and optionally report them.

Report when zlib support is missing.

The buffer is used for an additional pass of frame compression, so videos
can be coded without ever using it (and some are coded so indeed, e.g. in
Woodruff and the Schnibble of Azimuth game).