Fixes: 6798/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_G2M_fuzzer-5135899701542912

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpegSigned-off-by: Michael Niedermayer <michael@niedermayer.cc>

HALFQP should only be added to the inverse quantizer when the block is
coded with PQUANT. When PQUANT is equal to ALTPQUANT, the original test
for the addition of HALFQP fails. A negative value for mquant indicates
that the value was derived from VOPDQUANT.

Fixes #4372
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

correct forgotten v->blocks_off
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

HALFQP should only be added to the inverse quantizer when the block is
coded with PQUANT. See 8.1.3.8 in VC-1 spec.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

mspel indicates the use of bicubic interpolation. The check wrongly included
MVMODE MV_PMODE_1MV_HPEL as using bilinear interpolation.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

remove unused overlap smooting and loop filter
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

Add previously omitted overlap smooting and loop filtering for
frame/field-interlace pictures. For progressive pictures switch to the
re-implemented versions of overlap smooting and loop filtering.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The existing implementation only used vc1_put_signed_blocks_clamped for I and
BI frames. This rewritten version is also applicable to P frame both
progressive and frame/field-interlace.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The existing implementation did out-of-bounds reference pixel replication for
progressive reference frames. In interlaced reference frames both the even and
odd line on the horizontal edges need to be replicated.

Fixes #3262.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

According to VC-1 spec table 74, the last value in ff_vc1_dqscale should be
0x1041 instead of 0x1000.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

According to VC-1 spec 10.7.3.4, FIELDTX shall be set to the same type as the
motion vector for zero-coded blocks.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The loop filter for P interlace field pictures needs the reference field type.
For luma, the reference field type was already available. Store the reference
field type for color-difference as well.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The new loop filter needs additional MB properties to make its filtering
decisions.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The existing implementation did loop filtering for progressive
frames only. This rewritten version implements loop filtering for
all applicable frame types for both progessive and
frame/field-interlace.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The new overlap smooting filter smoothes image pixels stored in v->block.
Switch to v->block instead of s->block for storing decoded image pixels for P
frames. Additionally, we must take incrementing *_blk_idx out of the
vc1_put_signed_blocks_clamped function.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

The existing implementation did overlap smoothing for progressive
frames only. This rewritten version implements overlap smoothing
for all applicable frame types for both progessive and
frame/field-interlace.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

before:
419022 decicycles in assemble_fragment,    2047 runs,      1 skips

after:
104621 decicycles in assemble_fragment,    2045 runs,      3 skips

Benched with a 2 minutes long 720x480 DVD mpeg2 sample.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>

Avoids unnecessary allocs+copies and makes the code slightly simpler.
Signed-off-by: James Almer <jamrial@gmail.com>

Also fixes a bug where EOS buffer was sent with incorrect
pts when not using surface generation.
Signed-off-by: Matthieu Bouron <matthieu.bouron@gmail.com>
Signed-off-by: Aman Gupta <aman@tmm1.net>

Signed-off-by: Matthieu Bouron <matthieu.bouron@gmail.com>
Signed-off-by: Jan Ekström <jeebjp@gmail.com>
Signed-off-by: Aman Gupta <aman@tmm1.net>

As of 2a0eb868, ff_mediacodec_dec_is_flushing() only returns
true in delay_flush mode. Make this more obvious by adding
delay_flush to the if statement.
Signed-off-by: Matthieu Bouron <matthieu.bouron@gmail.com>
Signed-off-by: Aman Gupta <aman@tmm1.net>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

also fixes: runtime error: index 1456 out of bounds for type 'int16_t [16]'

Found-by: durandal_1707
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

The crc flag is only stored since version 3 thus before this crcs do not
work. We increase the version as needed same as we do with pix_fmts
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Interlaced frame pictures do not contain the MVMODE or MVMODE2 bitstream
element. Trying to parse this element and passing a nonzero value to the
hardware decoder results in small inaccuracies in the decoded picture.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This reverts commit 62f5c9d6,
which was pushed a bit prematurely.

Silences a warning:
libavformat/os_support.c:49:12: warning: 'inet_aton' defined but not used

Reported-by: Reino Wijnsma

Should be useful for muxers that require values as defined in the
vpcc atom but don't need to write the atom itself.
Signed-off-by: James Almer <jamrial@gmail.com>

and checkasm test

avcodec/prores_ks : use official quant_matrix (for proxy and xq codec luma and chroma quant matrix is not the same)

disable the use of the official luma xq matrix for now (output
appear to be desaturate)

Ignore extra 12 bytes that belong to tag and size.
Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes #5489.
Signed-off-by: Paul B Mahol <onemda@gmail.com>

When use new decode APIs(avcodec_send_packet/avcodec_receive_frame),
don't need to setting the deprecated field refcounted_frames.
Reviewed-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Jun Zhao <mypopydev@gmail.com>