The old code had two bugs:

For audio filters, the format was not set.

For video filters, if several links reference the same format list,
the same format must be selected in the end. This is done by
setting formats->format_count to 1: the other links sharing
the reference will therefore have only one choice.
If the heuristic does not pick the first format, the selected format
must also be moved to the first position.