This partially reverts cf70ba37, since
it didn't take into account when rotation is 0, but there is another
valid operation (eg. translation) in the matrix.
Found-by: Michael Niedermayer <michaelni@gmx.at>