There is no reason to keep this intact when decoding failed, specially
as private_ref is supposed to always be NULL when a frame is returned to
the user.