With the current approach we only need to track using an unordered set as we can
still access the backing store pointer and length by the time we free the
backing store.

Reland:
The issue was fixed in 67b5a501.

BUG=chromium:619491, chromium:611688
LOG=N
R=ulan@chromium.org

This reverts commit 0e1eaec7.

Review-Url: https://codereview.chromium.org/2109913003
Cr-Commit-Position: refs/heads/master@{#37399}