Add a method to gsutil recipe_module to disable multiprocessing on mac
In https://crbug.com/1327371, we started encountering hanging gsutil calls. These were triggered under very specific circumstances, but were essentially due to a multi-processing bug in python on MacOS: https://bugs.python.org/issue33725 When running gsutil on mac, it explicitly suggests disabling multi-processing for this reason: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331 So this CL simply puts that suggestion into practice. There are two options for doing so: adding a line to an active Boto file, or adding a "-o" arg to every gsutil invocation. Since chromium has multiple copies of gsutil and multiple different locations where `gsutil` is invoked, the latter option is considered intractable. Instead, we choose the Boto file option by adding a context manager to the gsutil recipe module that will insert a custom Boto file into the environment that only disables multi-threading. Due to the fact that the BOTO_PATH env var takes multiple paths, we can safely incorporate both our custom Boto along with LUCI's per-build Boto file in the same gsutil invocation. And when the context manager exits, the environment should revert back to its original state. Chromium incorporates this method in crrev.com/c/3662023. See it take effect during the "gclient runhooks" step in this led build: https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com Bug: 1327371 Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949Reviewed-by:Josip Sokcevic <sokcevic@google.com> Commit-Queue: Ben Pastene <bpastene@chromium.org>
Showing
Please
register
or
sign in
to comment