This logic mirrors what KeyedLoadICs are doing already.

BUG=chromium:602184
LOG=n

Review URL: https://codereview.chromium.org/1876793003

Cr-Commit-Position: refs/heads/master@{#35397}