Encoder output is delayed by several frames, so we keep a queue of input
frame timing info to match up with corresponding output packets.