• Martin Storsjö's avatar
    checkasm: aacpsdsp: Tolerate extra intermediate precision in stereo_interpolate · aad0e26f
    Martin Storsjö authored
    The stereo_interpolate functions add h_step to the values h
    BUF_SIZE times. Within the stereo_interpolate C functions, the
    values h (h0-h3, h00-h13) are declared as local float variables,
    but the compiler is free to keep them in a register with extra
    precision.
    
    If the accumulation is rounded to 32 bit float precision after
    each step, the less significant bits of h_step end up ignored
    and the sum can deviate, affecting the end result more than
    the currently set EPS.
    
    By clearing the log2(BUF_SIZE) lower bits of h_step, we make sure
    that the accumulation shouldn't differ significantly, regardless
    of any extra precision in the accmulating register/variable.
    
    This fixes the aacpsdsp checkasm test when built with clang for
    mingw/x86_32.
    Signed-off-by: 's avatarMartin Storsjö <martin@martin.st>
    aad0e26f