The valid returned values are always at most 11bit.
Remove the previous check that assumed larger values plausible and
use a signed integer to check get_vlc2 return values.

CC: libav-stable@libav.org