This is currently always true, the assert protects against
future changes to the code breaking this assumtation
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>