The value of coeff_per_sb_select depends on the bit rate, not on
sub_sampling.
Also remove the calculation of tmp, no longer needed.

Fixes tickets #1868 and #742 (only audio part)
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>