Add a method to gsutil recipe_module to disable multiprocessing on mac

In https://crbug.com/1327371, we started encountering hanging gsutil
calls. These were triggered under very specific circumstances, but
were essentially due to a multi-processing bug in python on MacOS:
https://bugs.python.org/issue33725

When running gsutil on mac, it explicitly suggests disabling
multi-processing for this reason:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331

So this CL simply puts that suggestion into practice. There are two
options for doing so: adding a line to an active Boto file, or adding
a "-o" arg to every gsutil invocation. Since chromium has multiple
copies of gsutil and multiple different locations where `gsutil` is
invoked, the latter option is considered intractable.

Instead, we choose the Boto file option by adding a context manager
to the gsutil recipe module that will insert a custom Boto file into
the environment that only disables multi-threading. Due to the fact
that the BOTO_PATH env var takes multiple paths, we can safely
incorporate both our custom Boto along with LUCI's per-build Boto file
in the same gsutil invocation. And when the context manager exits, the
environment should revert back to its original state.

Chromium incorporates this method in crrev.com/c/3662023. See it take
effect during the "gclient runhooks" step in this led build:
https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com

Bug: 1327371
Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

recipes/README.recipes.md

@@ -32,6 +32,7 @@
   * [git_cl:examples/full](#recipes-git_cl_examples_full) (Python3 ✅)
   * [gitiles:examples/full](#recipes-gitiles_examples_full) (Python3 ✅)
   * [gitiles:tests/parse_repo_url](#recipes-gitiles_tests_parse_repo_url) (Python3 ✅)
+  * [gsutil:examples/custom_boto](#recipes-gsutil_examples_custom_boto) (Python3 ✅)
   * [gsutil:examples/full](#recipes-gsutil_examples_full) (Python3 ✅)
   * [osx_sdk:examples/full](#recipes-osx_sdk_examples_full) (Python3 ✅)
   * [presubmit:examples/full](#recipes-presubmit_examples_full) (Python3 ✅)
@@ -636,13 +637,13 @@ Returns a list of refs in the remote repository.
 Generates a Gitiles repo URL. See also parse_repo_url.
 ### *recipe_modules* / [gsutil](/recipes/recipe_modules/gsutil)
 
-[DEPS](/recipes/recipe_modules/gsutil/__init__.py#3): [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+[DEPS](/recipes/recipe_modules/gsutil/__init__.py#5): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step]
 
 PYTHON_VERSION_COMPATIBILITY: PY2+3
 
-#### **class [GSUtilApi](/recipes/recipe_modules/gsutil/api.py#9)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
+#### **class [GSUtilApi](/recipes/recipe_modules/gsutil/api.py#10)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
 
-— **def [\_\_call\_\_](/recipes/recipe_modules/gsutil/api.py#14)(self, cmd, name=None, use_retry_wrapper=True, version=None, parallel_upload=False, multithreaded=False, infra_step=True, \*\*kwargs):**
+— **def [\_\_call\_\_](/recipes/recipe_modules/gsutil/api.py#21)(self, cmd, name=None, use_retry_wrapper=True, version=None, parallel_upload=False, multithreaded=False, infra_step=True, \*\*kwargs):**
 
 A step to run arbitrary gsutil commands.
 
@@ -660,25 +661,37 @@ Args:
   * name (str) - Name of the step to use. Defaults to the first non-flag
       token in the cmd.
 
-— **def [cat](/recipes/recipe_modules/gsutil/api.py#108)(self, url, args=None, \*\*kwargs):**
+— **def [cat](/recipes/recipe_modules/gsutil/api.py#115)(self, url, args=None, \*\*kwargs):**
 
-— **def [copy](/recipes/recipe_modules/gsutil/api.py#122)(self, source_bucket, source, dest_bucket, dest, args=None, link_name='gsutil.copy', metadata=None, unauthenticated_url=False, \*\*kwargs):**
+&emsp; **@contextlib.contextmanager**<br>&mdash; **def [configure\_gsutil](/recipes/recipe_modules/gsutil/api.py#169)(self, \*\*kwargs):**
 
-&mdash; **def [download](/recipes/recipe_modules/gsutil/api.py#94)(self, bucket, source, dest, args=None, \*\*kwargs):**
+Temporarily configures the behavior of gsutil.
 
-&mdash; **def [download\_url](/recipes/recipe_modules/gsutil/api.py#101)(self, url, dest, args=None, \*\*kwargs):**
+For the duration of its context, this method will temporarily append a
+custom Boto file to the BOTO_PATH env var without overwriting bbagent's
+BOTO_CONFIG. See https://cloud.google.com/storage/docs/boto-gsutil for
+possible configurations.
 
-&emsp; **@property**<br>&mdash; **def [gsutil\_py\_path](/recipes/recipe_modules/gsutil/api.py#10)(self):**
+Args:
+  kwargs: Every keyword arg is treated as config line in the temp Boto file.
+
+&mdash; **def [copy](/recipes/recipe_modules/gsutil/api.py#129)(self, source_bucket, source, dest_bucket, dest, args=None, link_name='gsutil.copy', metadata=None, unauthenticated_url=False, \*\*kwargs):**
+
+&mdash; **def [download](/recipes/recipe_modules/gsutil/api.py#101)(self, bucket, source, dest, args=None, \*\*kwargs):**
+
+&mdash; **def [download\_url](/recipes/recipe_modules/gsutil/api.py#108)(self, url, dest, args=None, \*\*kwargs):**
+
+&emsp; **@property**<br>&mdash; **def [gsutil\_py\_path](/recipes/recipe_modules/gsutil/api.py#17)(self):**
 
-&mdash; **def [list](/recipes/recipe_modules/gsutil/api.py#141)(self, url, args=None, \*\*kwargs):**
+&mdash; **def [list](/recipes/recipe_modules/gsutil/api.py#148)(self, url, args=None, \*\*kwargs):**
 
-&mdash; **def [remove\_url](/recipes/recipe_modules/gsutil/api.py#155)(self, url, args=None, \*\*kwargs):**
+&mdash; **def [remove\_url](/recipes/recipe_modules/gsutil/api.py#162)(self, url, args=None, \*\*kwargs):**
 
-&mdash; **def [signurl](/recipes/recipe_modules/gsutil/api.py#148)(self, private_key_file, bucket, dest, args=None, \*\*kwargs):**
+&mdash; **def [signurl](/recipes/recipe_modules/gsutil/api.py#155)(self, private_key_file, bucket, dest, args=None, \*\*kwargs):**
 
-&mdash; **def [stat](/recipes/recipe_modules/gsutil/api.py#115)(self, url, args=None, \*\*kwargs):**
+&mdash; **def [stat](/recipes/recipe_modules/gsutil/api.py#122)(self, url, args=None, \*\*kwargs):**
 
-&mdash; **def [upload](/recipes/recipe_modules/gsutil/api.py#77)(self, source, bucket, dest, args=None, link_name='gsutil.upload', metadata=None, unauthenticated_url=False, \*\*kwargs):**
+&mdash; **def [upload](/recipes/recipe_modules/gsutil/api.py#84)(self, source, bucket, dest, args=None, link_name='gsutil.upload', metadata=None, unauthenticated_url=False, \*\*kwargs):**
 ### *recipe_modules* / [osx\_sdk](/recipes/recipe_modules/osx_sdk)
 
 [DEPS](/recipes/recipe_modules/osx_sdk/__init__.py#7): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/version][recipe_engine/recipe_modules/version]
@@ -1094,6 +1107,13 @@ PYTHON_VERSION_COMPATIBILITY: PY2+3
 PYTHON_VERSION_COMPATIBILITY: PY2+3
 
 &mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#14)(api):**
+### *recipes* / [gsutil:examples/custom\_boto](/recipes/recipe_modules/gsutil/examples/custom_boto.py)
+
+[DEPS](/recipes/recipe_modules/gsutil/examples/custom_boto.py#10): [gsutil](#recipe_modules-gsutil), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties]
+
+PYTHON_VERSION_COMPATIBILITY: PY2+3
+
+&mdash; **def [RunSteps](/recipes/recipe_modules/gsutil/examples/custom_boto.py#21)(api, boto_configs):**
 ### *recipes* / [gsutil:examples/full](/recipes/recipe_modules/gsutil/examples/full.py)
 
 [DEPS](/recipes/recipe_modules/gsutil/examples/full.py#7): [gsutil](#recipe_modules-gsutil), [recipe\_engine/path][recipe_engine/recipe_modules/path]

recipes/recipe_modules/gsutil/__init__.py

+from PB.recipe_modules.depot_tools.gsutil import properties
+
 PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
 
 DEPS = [
+  'recipe_engine/context',
+  'recipe_engine/file',
   'recipe_engine/path',
+  'recipe_engine/platform',
   'recipe_engine/step',
 ]
+
+ENV_PROPERTIES = properties.EnvProperties

recipes/recipe_modules/gsutil/api.py

@@ -2,11 +2,18 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
+import contextlib
 import re
 
 from recipe_engine import recipe_api
 
 class GSUtilApi(recipe_api.RecipeApi):
+
+  def __init__(self, env_properties, *args, **kwargs):
+    super(GSUtilApi, self).__init__(*args, **kwargs)
+    self._boto_config_path = env_properties.BOTO_CONFIG
+    self._boto_path = env_properties.BOTO_PATH
+
   @property
   def gsutil_py_path(self):
     return self.repo_resource('gsutil.py')
@@ -159,6 +166,58 @@ class GSUtilApi(recipe_api.RecipeApi):
     name = kwargs.pop('name', 'remove')
     return self(cmd, name, **kwargs)
 
+  @contextlib.contextmanager
+  def configure_gsutil(self, **kwargs):
+    """Temporarily configures the behavior of gsutil.
+
+    For the duration of its context, this method will temporarily append a
+    custom Boto file to the BOTO_PATH env var without overwriting bbagent's
+    BOTO_CONFIG. See https://cloud.google.com/storage/docs/boto-gsutil for
+    possible configurations.
+
+    Args:
+      kwargs: Every keyword arg is treated as config line in the temp Boto file.
+    """
+    if self.m.platform.is_mac:
+      # Due to https://bugs.python.org/issue33725, using gsutil to download
+      # sufficiently large files on MacOS has been seen to hang indefinitely,
+      # and disabling multi-processing avoids that hang.
+      kwargs.setdefault('parallel_process_count', '1')
+    if not kwargs:
+      yield
+      return
+
+    # If neither BOTO_CONFIG nor BOTO_PATH are set, gsutil looks at default
+    # locations (/etc/boto.cfg and ~/.boto). So give up in that case just to
+    # avoid the hassle of incorporating all the defaults. ~All LUCI builds
+    # should at least be setting BOTO_CONFIG.
+    if not self._boto_config_path and not self._boto_path:
+      yield
+      return
+    custom_boto_path = self.m.path.mkstemp(prefix='custom_boto_')
+    contents = [
+        '# Generated by $depot_tools.recipe_modules.gsutil',
+        # https://cloud.google.com/storage/docs/boto-gsutil seems to indicate
+        # that the section headers are important. So certain config lines may
+        # not work unless they show up under the appropriate header.
+        '[GSUtil]',
+    ]
+    for k, v in kwargs.items():
+      contents.append('%s = %s' % (k, str(v)))
+    self.m.file.write_text(
+        'write temp Boto file', custom_boto_path, '\n'.join(contents))
+    # BOTO_CONFIG can only point to one file; BOTO_PATH can point to multiple,
+    # each joined by ':'. If BOTO_CONFIG is set, BOTO_PATH is ignored.
+    if self._boto_config_path:
+      custom_boto_path = (
+          self._boto_config_path + ':' + self.m.path.abspath(custom_boto_path))
+    elif self._boto_path:
+      custom_boto_path = (
+          self._boto_path + ':' + self.m.path.abspath(custom_boto_path))
+    with self.m.context(
+        env={'BOTO_PATH': custom_boto_path, 'BOTO_CONFIG': None}):
+      yield
+
   def _generate_metadata_args(self, metadata):
     result = []
     if metadata:

recipes/recipe_modules/gsutil/examples/custom_boto.py

+# Copyright 2013 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from recipe_engine import post_process
+from recipe_engine.recipe_api import Property
+
+PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
+
+DEPS = [
+  'gsutil',
+  'recipe_engine/platform',
+  'recipe_engine/properties',
+]
+
+PROPERTIES = {
+  'boto_configs': Property(default={}, kind=dict),
+}
+
+
+def RunSteps(api, boto_configs):
+  with api.gsutil.configure_gsutil(**boto_configs):
+    api.gsutil(['cp', 'gs://some/gs/path', '/some/local/path'])
+
+
+def GenTests(api):
+  yield api.test(
+      'no_args',
+      api.platform('linux', 64),
+      api.post_process(post_process.DropExpectation),
+  )
+
+  yield api.test(
+      'no_env',
+      api.properties(boto_configs={'some_config': 'some_val'}),
+      api.post_process(post_process.DropExpectation),
+  )
+
+  yield api.test(
+      'mac',
+      api.platform('mac', 64),
+      api.post_process(post_process.DropExpectation),
+  )
+
+  yield api.test(
+      'with_boto_config',
+      api.properties(boto_configs={'some_config': 'some_val'}),
+      api.properties.environ(
+          BOTO_CONFIG='/some/boto/config'
+      ),
+      api.post_check(lambda check, steps: \
+          check(steps['gsutil cp'].env['BOTO_CONFIG'] is None)),
+      api.post_check(lambda check, steps: \
+          check('/some/boto/config' in steps['gsutil cp'].env['BOTO_PATH'])),
+      api.post_process(post_process.DropExpectation),
+  )
+
+  yield api.test(
+      'with_boto_path',
+      api.properties(boto_configs={'some_config': 'some_val'}),
+      api.properties.environ(
+          BOTO_PATH='/some/boto/path'
+      ),
+      api.post_check(lambda check, steps: \
+          check(steps['gsutil cp'].env['BOTO_CONFIG'] is None)),
+      api.post_check(lambda check, steps: \
+          check('/some/boto/path' in steps['gsutil cp'].env['BOTO_PATH'])),
+      api.post_process(post_process.DropExpectation),
+  )

recipes/recipe_modules/gsutil/properties.proto

+// Copyright 2022 The LUCI Authors. All rights reserved.
+// Use of this source code is governed under the Apache License, Version 2.0
+// that can be found in the LICENSE file.
+syntax = "proto3";
+
+package recipe_modules.depot_tools.gsutil;
+
+message EnvProperties {
+    // Env vars that gsutil uses for both authentication and configuration.
+    string BOTO_CONFIG = 1;
+    string BOTO_PATH = 2;
+}