The documentation states that av_image_check_size() will return a
negative value in case of error, while the check is done on ret != 0.

Also return a proper error code rather than -1 in case the check fails.