This increases the PSNR slightly (about 0.1 dB) for trellis sizes
below 8, and gives equal PSNR for sizes above that.

Originally committed as revision 25769 to svn://svn.ffmpeg.org/ffmpeg/trunk