This avoids having to assign oformat, allows returning the
correct error code and allocates priv_data

Based on patch by: Mika Raento <mika.raento@elisa.fi>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>