Fixes: u263_b-frames_1.avi
Fixes part of Ticket1536

return -1 is used here as it is used in similar code in this function, I intend
to replace it by proper error codes in the whole function.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>