• Rostislav Pehlivanov's avatar
    aacenc_tns: rework the way coefficients are calculated · f20b6717
    Rostislav Pehlivanov authored
    This commit abandons the way the specifications state to
    quantize the coefficients, makes use of the new LPC float
    functions and is much better.
    
    The original way of converting non-normalized float samples
    to int32_t which out LPC system expects was wrong and it was
    wrong to assume the coefficients that are generated are also
    valid. It was essentially a full garbage-in, garbage-out
    system and it definitely shows when looking at spectrals
    and listening. The high frequencies were very overattenuated.
    The new LPC function performs the analysis directly.
    
    The specifications state to quantize the coefficients into
    four bit index values using an asin() function which of course
    had to have ugly ternary operators because the function turns
    negative if the coefficients are negative which when encoding
    causes invalid bitstream to get generated.
    
    This deviates from this by using the direct TNS tables, which
    are fairly small since you only have 4 bits at most for index
    values. The LPC values are directly quantized against the tables
    and are then used to perform filtering after the requantization,
    which simply fetches the array values.
    
    The end result is that TNS works much better now and doesn't
    attenuate anything but the actual signal, e.g. TNS removes
    quantization errors and does it's job correctly now.
    
    It might be enabled by default soon since it doesn't hurt and
    helps reduce nastyness at low bitrates.
    Signed-off-by: 's avatarRostislav Pehlivanov <atomnuker@gmail.com>
    f20b6717
aacenc_tns.h 1.28 KB