The only thing the demuxer needs is the sample rate to set the timebase,
which can be simply read with AV_RB32.