This allows specifying more dither algorithms without using up flags and
without ambiguities.

Also initialize the new field based on the flags and use it.
Note, improving the logic of the checks is left to subsequent
commits, this here only switches from flags to enum.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>