This gets rid of a number of warnings about casts discarding
qualifiers from the pointer target, present since 7ebfb466.
Signed-off-by: Martin Storsjö <martin@martin.st>