The use of ff_inverse speeds up slightly arches + compilers that
do not provide a division faster than the whole machinery, such
as ppc32 + gcc4.7, but has operational limits.

Drop the always-enable assert and provide a fallback.