Those sets are to be released on the main thread without concurrent
accesses. Making this explicit will give TSAN a chance to help us once
the surrounding code changes.

Bug: 
Change-Id: Ia73754caafbeec385d4c922fb8140e3e64f7378c
Reviewed-on: https://chromium-review.googlesource.com/541375Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#46025}