This can be made more efficient, but first and the main goal of this change is to
store it at all
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>