Files with invalid VUI are now rejected like
other invalid SPS are.

Fixes issue1231.

Originally committed as revision 19335 to svn://svn.ffmpeg.org/ffmpeg/trunk