This fixes a TODO item and unifies both decoders structures
It also fixes undefined behavior due to aliasing violations

I choose 2 fields instead of a union because mistakely using the
wrong type with a union will lead to hard to debug "wrong output"
while with 2 fields mistakely using the wrong type will crash
with a null pointer derefernce which is much easier to debug
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>