Fixes: af94b3a3d26586c08f557cafe8246251_signal_sigsegv_7ffff713351a_343_XFMode.ASF with 2097152 alloc limit

Found-by: Samuel Groß, Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>