In rare corner cases it could still fail an assert on sf_diff due
to failure to update prev_sf in some code paths. Fix that case.