This limits the bugs, speedloss and extra memory allocation to the case when
optimal tables are needed.
Fixes regressions with slice multi-threading
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>