AVERROR_NOFMT in case of invalid / unknown packet size.

Originally committed as revision 22624 to svn://svn.ffmpeg.org/ffmpeg/trunk