Commit be1bafc3 authored by Ronald S. Bultje's avatar Ronald S. Bultje Committed by Mans Rullgard

swscale: fix overflows in output of RGB48 pixels.

For certain types of filters where the intermediate sum of coefficients
can go above the fixed-point equivalent of 1.0 in the middle of a filter,
the sum of a 31-bit calculation can overflow in both directions and can
thus not be represented in a 32-bit signed or unsigned integer. To work
around this, we subtract 0x40000000 from a signed integer base, so that
we're halfway signed/unsigned, which makes it fit even if it overflows.
After the filter finishes, we add the scaled bias back after a shift.

We use the same trick for 16-bit bpc YUV output routines.
Signed-off-by: 's avatarMans Rullgard <mans@mansr.com>
parent 5d8122db
...@@ -754,8 +754,8 @@ yuv2rgb48_X_c_template(SwsContext *c, const int16_t *lumFilter, ...@@ -754,8 +754,8 @@ yuv2rgb48_X_c_template(SwsContext *c, const int16_t *lumFilter,
for (i = 0; i < (dstW >> 1); i++) { for (i = 0; i < (dstW >> 1); i++) {
int j; int j;
int Y1 = 0; int Y1 = -0x40000000;
int Y2 = 0; int Y2 = -0x40000000;
int U = -128 << 23; // 19 int U = -128 << 23; // 19
int V = -128 << 23; int V = -128 << 23;
int R, G, B; int R, G, B;
...@@ -771,7 +771,9 @@ yuv2rgb48_X_c_template(SwsContext *c, const int16_t *lumFilter, ...@@ -771,7 +771,9 @@ yuv2rgb48_X_c_template(SwsContext *c, const int16_t *lumFilter,
// 8bit: 12+15=27; 16-bit: 12+19=31 // 8bit: 12+15=27; 16-bit: 12+19=31
Y1 >>= 14; // 10 Y1 >>= 14; // 10
Y1 += 0x10000;
Y2 >>= 14; Y2 >>= 14;
Y2 += 0x10000;
U >>= 14; U >>= 14;
V >>= 14; V >>= 14;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment