There is no need to duplicate the input reference, since a filter should
not pass along a reference which is later modified. If this happens the
filter passing the reference should be fixed.

Also improve consistency with start_frame() of settb, allowing a pending
factorization.