AVERROR(EINVAL) rather than -1 in case of unsuccess.

Originally committed as revision 24521 to svn://svn.ffmpeg.org/ffmpeg/trunk