For video, frame_number tracks the number of frames sent to the encoder.
So it should be incremented when we submit a frame, not when we get a
packet back.