I realized that the bug was not in the imdct routine but in the
get_transform_coeffs.
Fixed it.
Code now uses the ffmpeg's imdct routines.
All the mplayer's ac3 samples are decoded
successfully.
Also improved downmixing.
Now all the downmixing coeffcients for channels
are normalized such that the sum of coefficients
used to construct the output for single channel
never exceeds 1.0.

Originally committed as revision 9659 to svn://svn.ffmpeg.org/ffmpeg/trunk