Use correct value for range

The current range value causes an underflow
when negated and pushes anything less than zero
to the minimum.
parent 768261e5
......@@ -895,7 +895,7 @@ static void lms_update(WmallDecodeCtx *s, int ich, int ilms, int16_t input, int1
{
int16_t icoef;
int recent = s->cdlms[ich][ilms].recent;
int16_t range = 1 << (s->bits_per_sample - 1);
int16_t range = (1 << s->bits_per_sample - 1) - 1;
int bps = s->bits_per_sample > 16 ? 4 : 2; // bytes per sample
if (input > pred) {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment