This reverts commit 04aa09c4
and reintroduces 0ff5567a that
was temporarily reverted due to minor regressions.

It also reverts e5bce8b4 that fixed FATE refs.
The fate-ffm change is caused by field_order now being set
on the output format because the first frame arrives earlier.
The fate-mxf change is assumed to be the same.