The buffer is used for an additional pass of frame compression, so videos
can be coded without ever using it (and some are coded so indeed, e.g. in
Woodruff and the Schnibble of Azimuth game).