Also properly return AVERROR(ERANGE) in case of actual overflows.
Signed-off-by: Marton Balint <cus@passwd.hu>