This avoids divisions by zero later (and possibly assertions in
time base scaling), since an invalid rate_flag combined with an
invalid bitrate below could pass the mode combination test.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>