This reverts commit 4abfa387.

This commit broke playback of fragmented mp4 files with b-frames.
While investigating this, it turned out that the general framework
isn't ready for a PTS-based index yet. Revert this change until
a better thought out solution is in place.
Signed-off-by: Martin Storsjö <martin@martin.st>