Skip to content

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Contribute to GitLab
  • Sign in / Register
C
Chromium_Depot_tools
  • Project
    • Project
    • Details
    • Activity
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
  • Issues 0
    • Issues 0
    • List
    • Board
    • Labels
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Linshizhi
  • Chromium_Depot_tools
  • Repository

Switch branch/tag
  • chromium_depot_tools
  • recipes
  • recipe_modules
  • gsutil
  • api.py
Find file
BlameHistoryPermalink
  • Ben Pastene's avatar
    Add a method to gsutil recipe_module to disable multiprocessing on mac · 8d2d507a
    Ben Pastene authored May 25, 2022
    In https://crbug.com/1327371, we started encountering hanging gsutil
    calls. These were triggered under very specific circumstances, but
    were essentially due to a multi-processing bug in python on MacOS:
    https://bugs.python.org/issue33725
    
    When running gsutil on mac, it explicitly suggests disabling
    multi-processing for this reason:
    https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331
    
    So this CL simply puts that suggestion into practice. There are two
    options for doing so: adding a line to an active Boto file, or adding
    a "-o" arg to every gsutil invocation. Since chromium has multiple
    copies of gsutil and multiple different locations where `gsutil` is
    invoked, the latter option is considered intractable.
    
    Instead, we choose the Boto file option by adding a context manager
    to the gsutil recipe module that will insert a custom Boto file into
    the environment that only disables multi-threading. Due to the fact
    that the BOTO_PATH env var takes multiple paths, we can safely
    incorporate both our custom Boto along with LUCI's per-build Boto file
    in the same gsutil invocation. And when the context manager exits, the
    environment should revert back to its original state.
    
    Chromium incorporates this method in crrev.com/c/3662023. See it take
    effect during the "gclient runhooks" step in this led build:
    https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com
    
    Bug: 1327371
    Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6
    Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949Reviewed-by: 's avatarJosip Sokcevic <sokcevic@google.com>
    Commit-Queue: Ben Pastene <bpastene@chromium.org>
    8d2d507a
api.py 10.1 KB
EditWeb IDE

Replace api.py

Attach a file by drag & drop or click to upload


Cancel
A new branch will be created in your fork and a new merge request will be started.