Commit 97957236 authored by Ganesh Ajjanagadde's avatar Ganesh Ajjanagadde Committed by Michael Niedermayer

avcodec/ac3enc: fix undefined negative left shift

This should fix the undefined behavior reported in:
https://trac.ffmpeg.org/ticket/4727.

I can reproduce this at runtime: simply stick in an abort call in
asym_quant to check if c < 0 and run FATE. I don't know ac3 so I can't
confirm if negative coefficients are intentional, but at the moment they
clearly are according to FATE.

This resolves the undefined behavior. Tested with FATE.
Signed-off-by: 's avatarGanesh Ajjanagadde <gajjanagadde@gmail.com>
Signed-off-by: 's avatarMichael Niedermayer <michael@niedermayer.cc>
parent c4e23ca8
......@@ -1183,7 +1183,7 @@ static inline int asym_quant(int c, int e, int qbits)
{
int m;
c = (((c << e) >> (24 - qbits)) + 1) >> 1;
c = (((c * (1<<e)) >> (24 - qbits)) + 1) >> 1;
m = (1 << (qbits-1));
if (c >= m)
c = m - 1;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment