Fixes potential security issue in case of running out of memory

Found-by: ubitux
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>