Commit 20a30077 authored by Ganesh Ajjanagadde's avatar Ganesh Ajjanagadde

avutil/mathematics: correct documentation for av_gcd

av_gcd is now always defined regardless of input. This documents this
change in the "documented API". Two benefits (closely related):
1. The function is robust, and there is no need to worry about INT64_MIN, etc.

2. Clients of av_gcd, like av_reduce, can now be made fully correct. Currently,
av_reduce can trigger undefined behavior if e.g num is INT64_MIN due to
integer overflow in the FFABS. Furthermore, this undefined behavior is
completely undocumented, and could be a fuzzer's paradise. The FFABS was needed in the past as
av_gcd was undefined for negative inputs. In order to make av_reduce
robust, it is essential to guarantee that av_gcd works for all int64_t.
Reviewed-by: 's avatarMichael Niedermayer <michael@niedermayer.cc>
Signed-off-by: 's avatarGanesh Ajjanagadde <gajjanagadde@gmail.com>
parent e8e7eb15
...@@ -77,9 +77,10 @@ enum AVRounding { ...@@ -77,9 +77,10 @@ enum AVRounding {
}; };
/** /**
* Return the greatest common divisor of a and b. * Compute the greatest common divisor of a and b.
* If both a and b are 0 or either or both are <0 then behavior is *
* undefined. * @return gcd of a and b up to sign; if a >= 0 and b >= 0, return value is >= 0;
* if a == 0 and b == 0, returns 0.
*/ */
int64_t av_const av_gcd(int64_t a, int64_t b); int64_t av_const av_gcd(int64_t a, int64_t b);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment