Fixes assertion failures in trellis encoder
Fixes valgrind warnings in trellis encoder
Fixes CID732257
Fixes CID732256
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>