This fixes decoding, broken since 7e350379.

This is similar to what was done for the normal mp3 decoder in
f4a86bc9.
Signed-off-by: Martin Storsjö <martin@martin.st>