An AVIStream (intended to be used as private data for an AVStream) would
leak in this scenario.

Also return a more fitting error code instead of AVERROR_INVALIDDATA.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>