Check memory earlier, check one more allocation and clean up on error.

CC: libav-stable@libav.org
Bug-Id: CID 1257773