fixing encoding at high QPs (was broken after the dct accuracy increase due to too small inverse table)

Originally committed as revision 901 to svn://svn.ffmpeg.org/ffmpeg/trunk