The reference map is never used in such cases, but we accidently copied
it anyway. This could cause crashes if this map has not yet been
allocated. Fixes trac ticket 3188.
Reviewed-by: Clément Bœsch <u@pkh.me>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>