Use just the pointer to the head of the buffer pool.

This will allow sharing the code with avplay.