This also reduces the amount of memory needed
Fixes Ticket4672

The new code seems slightly faster as well, probably due to better cache usage
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>