Commit c9fe0caf authored by Guillaume Martres's avatar Guillaume Martres Committed by Michael Niedermayer

hevc: clip pixels when transquant bypass is used

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

This is a more proper fix than 5856bca3

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1
Signed-off-by: 's avatarMichael Niedermayer <michaelni@gmx.at>
parent 65801040
......@@ -52,7 +52,7 @@ static void FUNC(transquant_bypass4x4)(uint8_t *_dst, int16_t *coeffs,
for (y = 0; y < 4; y++) {
for (x = 0; x < 4; x++) {
dst[x] += *coeffs;
dst[x] = av_clip_pixel(dst[x] + *coeffs);
coeffs++;
}
dst += stride;
......@@ -69,7 +69,7 @@ static void FUNC(transquant_bypass8x8)(uint8_t *_dst, int16_t *coeffs,
for (y = 0; y < 8; y++) {
for (x = 0; x < 8; x++) {
dst[x] += *coeffs;
dst[x] = av_clip_pixel(dst[x] + *coeffs);
coeffs++;
}
dst += stride;
......@@ -86,7 +86,7 @@ static void FUNC(transquant_bypass16x16)(uint8_t *_dst, int16_t *coeffs,
for (y = 0; y < 16; y++) {
for (x = 0; x < 16; x++) {
dst[x] += *coeffs;
dst[x] = av_clip_pixel(dst[x] + *coeffs);
coeffs++;
}
dst += stride;
......@@ -103,7 +103,7 @@ static void FUNC(transquant_bypass32x32)(uint8_t *_dst, int16_t *coeffs,
for (y = 0; y < 32; y++) {
for (x = 0; x < 32; x++) {
dst[x] += *coeffs;
dst[x] = av_clip_pixel(dst[x] + *coeffs);
coeffs++;
}
dst += stride;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment