some PSNR/bitrate improvement for adaptive quant

Originally committed as revision 7346 to svn://svn.ffmpeg.org/ffmpeg/trunk