Otherwise the decoder will throw "Missing header" errors when the
packets are sent for decoding.

This is similar to 89a420b7.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>