return AVERROR(ENOMEM) if alloc failed, set count after alloc, use variable when reallocing, based on patch by Netgem

Originally committed as revision 17509 to svn://svn.ffmpeg.org/ffmpeg/trunk