• Ganesh Ajjanagadde's avatar
    avutil/mathematics: make av_gcd more robust · b7fb7c45
    Ganesh Ajjanagadde authored
    This ensures that no undefined behavior is invoked, while retaining
    identical return values in all cases and at no loss of performance
    (identical asm on clang and gcc).
    Essentially, this patch exchanges undefined behavior with implementation
    defined behavior, a strict improvement.
    
    Rationale:
    1. The ideal solution is to have the return type a uint64_t. This
    unfortunately requires an API change.
    2. The only pathological behavior happens if both arguments are
    INT64_MIN, to the best of my knowledge. In such a case, the
    implementation defined behavior is invoked in the sense that UINT64_MAX
    is interpreted as INT64_MIN, which any reasonable implementation will
    do. In any case, any usage where both arguments are INT64_MIN is a
    fuzzer anyway.
    3. Alternatives of checking, etc require branching and lose performance
    for no concrete gain - no client cares about av_gcd's actual value when
    both args are INT64_MIN. Even if it did, on sane platforms (e.g all the
    ones FFmpeg cares about), it produces a correct gcd, namely INT64_MIN.
    Reviewed-by: 's avatarMichael Niedermayer <michael@niedermayer.cc>
    Signed-off-by: 's avatarGanesh Ajjanagadde <gajjanagadde@gmail.com>
    b7fb7c45
mathematics.c 5.53 KB