This allows removing a special case for the fixed point decoder and will
make error checks simpler
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>