Previously these were left in an inconsistent state.

Pointed out by Michael Niedermayer.
Signed-off-by: Martin Storsjö <martin@martin.st>