The 4-tap filters should only access one row/column before the
reference block.
Signed-off-by: Mans Rullgard <mans@mansr.com>