The color fields were moved to another struct, and a way to propagate
timestamps and other input metadata was introduced, so the packet fifo
can be removed.
Signed-off-by: James Almer <jamrial@gmail.com>