Print error when max resync size is reached and return EAGAIN instead of
INVALIDDATA, so user can retry if wanted.

Originally committed as revision 20286 to svn://svn.ffmpeg.org/ffmpeg/trunk