fix b frames + -mbd rd without -psnr (yes that was seriously buggy and probably hasnt been noticed as everyone who tested stuff used -psnr)

Originally committed as revision 7151 to svn://svn.ffmpeg.org/ffmpeg/trunk