• Bruce Dawson's avatar
    Mitigate Python 3 multiprocessing bug on Windows · 8254f06b
    Bruce Dawson authored
    The multiprocessing module on Windows has a bug where if you ask for
    more than 60 child processes then it will hang. This is related to the
    MAXIMUM_WAIT_OBJECTS (64) limit of WaitForMultipleObjects. Other sources
    have listed the multiprocessing limit as being 61, or have said that the
    the maximum number of objects that can be waited on is actually 63, but
    those details don't really matter.
    
    The original fix for this class of issues was crrev.com/c/2785844. This
    change extends those fixes to depot_tools, which was missed last year.
    This change also updates how PyLint is called by further limiting the
    number of jobs to the number of files being processed divided by 10.
    This is because there is a significant cost to creating PyLint
    subprocesses - each takes about 0.5 s on my test machine. So there needs
    to be enough parallelism to justify this.
    
    Patches for PyLint and a bug for cpython are planned.
    
    This will stop PyLint from hanging during presubmits on many-core
    machines. The command used to reproduce the hangs and validate the fix
    was:
    
      git cl presubmit -v --force --files "chrome/test/mini_installer/*.py"
    
    Prior to this change this command would use (on my many-core test
    machine) 96 processes and would hang. How it uses just two processes
    because there are only 16 files to analyze.
    
    Output before:
      Pylint (16 files using ['--disable=cyclic-import'] on 96 cores)
    Output after:
      Pylint (16 files using ['--disable=cyclic-import'] on 2 processes)
    
    This is actually not quite true because the hang would prevent the
    old message from being displayed.
    
    Bug: 1190269, 1336854
    Change-Id: Ie82baf91df4364a92eb664a00cf9daf167e0a548
    Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3711282Reviewed-by: 's avatarGavin Mak <gavinmak@google.com>
    Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
    8254f06b
Name
Last commit
Last update
bootstrap Loading commit data...
fetch_configs Loading commit data...
git-templates Loading commit data...
infra Loading commit data...
man Loading commit data...
python-bin Loading commit data...
python2-bin Loading commit data...
recipes Loading commit data...
testing_support Loading commit data...
tests Loading commit data...
third_party Loading commit data...
win_toolchain Loading commit data...
zsh-goodies Loading commit data...
.cipd_impl.ps1 Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.isort.cfg Loading commit data...
.style.yapf Loading commit data...
.vpython Loading commit data...
.vpython3 Loading commit data...
CROS_OWNERS Loading commit data...
DIR_METADATA Loading commit data...
GOMA_OWNERS Loading commit data...
LICENSE Loading commit data...
LUCI_OWNERS Loading commit data...
OWNERS Loading commit data...
PRESUBMIT.py Loading commit data...
README.gclient.md Loading commit data...
README.git-cl.md Loading commit data...
README.md Loading commit data...
WATCHLISTS Loading commit data...
auth.py Loading commit data...
autoninja Loading commit data...
autoninja.bat Loading commit data...
autoninja.py Loading commit data...
bb Loading commit data...
bb.bat Loading commit data...
black Loading commit data...
bootstrap_python3 Loading commit data...
breakpad.py Loading commit data...
cbuildbot Loading commit data...
cipd Loading commit data...
cipd.bat Loading commit data...
cipd_bin_setup.bat Loading commit data...
cipd_bin_setup.sh Loading commit data...
cipd_client_version Loading commit data...
cipd_client_version.digests Loading commit data...
cipd_manifest.txt Loading commit data...
cipd_manifest.versions Loading commit data...
cit Loading commit data...
cit.bat Loading commit data...
cit.py Loading commit data...
clang-format Loading commit data...
clang-format.bat Loading commit data...
clang_format.py Loading commit data...
clang_format_merge_driver Loading commit data...
clang_format_merge_driver.bat Loading commit data...
clang_format_merge_driver.py Loading commit data...
codereview.settings Loading commit data...
compile_single_file Loading commit data...
compile_single_file.bat Loading commit data...
compile_single_file.py Loading commit data...
cpplint.bat Loading commit data...
cpplint.py Loading commit data...
cpplint_chromium.py Loading commit data...
cros Loading commit data...
cros_sdk Loading commit data...
crosjobs Loading commit data...
detect_host_arch.py Loading commit data...
dirmd Loading commit data...
dirmd.bat Loading commit data...
download_from_google_storage Loading commit data...
download_from_google_storage.bat Loading commit data...
download_from_google_storage.py Loading commit data...
ensure_bootstrap Loading commit data...
fetch Loading commit data...
fetch.bat Loading commit data...
fetch.py Loading commit data...
fix_encoding.py Loading commit data...
gclient Loading commit data...
gclient-new-workdir.py Loading commit data...
gclient.bat Loading commit data...
gclient.py Loading commit data...
gclient_completion.sh Loading commit data...
gclient_eval.py Loading commit data...
gclient_paths.py Loading commit data...
gclient_scm.py Loading commit data...
gclient_utils.py Loading commit data...
gerrit_client.py Loading commit data...
gerrit_util.py Loading commit data...
git-cache Loading commit data...
git-cl Loading commit data...
git-crrev-parse Loading commit data...
git-drover Loading commit data...
git-find-releases Loading commit data...
git-footers Loading commit data...
git-freeze Loading commit data...
git-gs Loading commit data...
git-hyper-blame Loading commit data...
git-map Loading commit data...
git-map-branches Loading commit data...
git-mark-merge-base Loading commit data...
git-migrate-default-branch Loading commit data...
git-nav-downstream Loading commit data...
git-nav-upstream Loading commit data...
git-new-branch Loading commit data...
git-number Loading commit data...
git-rebase-update Loading commit data...
git-rename-branch Loading commit data...
git-reparent-branch Loading commit data...
git-retry Loading commit data...
git-runhooks Loading commit data...
git-squash-branch Loading commit data...
git-thaw Loading commit data...
git-upstream-diff Loading commit data...
git_cache.py Loading commit data...
git_cl.py Loading commit data...
git_cl_completion.sh Loading commit data...
git_common.py Loading commit data...
git_dates.py Loading commit data...
git_drover.py Loading commit data...
git_find_releases.py Loading commit data...
git_footers.py Loading commit data...
git_freezer.py Loading commit data...
git_hyper_blame.py Loading commit data...
git_map.py Loading commit data...
git_map_branches.py Loading commit data...
git_mark_merge_base.py Loading commit data...
git_migrate_default_branch.py Loading commit data...
git_nav_downstream.py Loading commit data...
git_new_branch.py Loading commit data...
git_number.py Loading commit data...
git_rebase_update.py Loading commit data...
git_rename_branch.py Loading commit data...
git_reparent_branch.py Loading commit data...
git_retry.py Loading commit data...
git_squash_branch.py Loading commit data...
git_upstream_diff.py Loading commit data...
gn Loading commit data...
gn.bat Loading commit data...
gn.py Loading commit data...
goma_auth Loading commit data...
goma_auth.bat Loading commit data...
goma_ctl Loading commit data...
goma_ctl.bat Loading commit data...
gsutil.py Loading commit data...
gsutil.py.bat Loading commit data...
gsutil.vpython3 Loading commit data...
isort Loading commit data...
led Loading commit data...
led.bat Loading commit data...
lockfile.py Loading commit data...
luci-auth Loading commit data...
luci-auth.bat Loading commit data...
lucicfg Loading commit data...
lucicfg.bat Loading commit data...
mac_toolchain Loading commit data...
metrics.README.md Loading commit data...
metrics.py Loading commit data...
metrics_utils.py Loading commit data...
my_activity.py Loading commit data...
ninja Loading commit data...
ninja-linux64 Loading commit data...
ninja-mac Loading commit data...
ninja.exe Loading commit data...
ninjalog.README.md Loading commit data...
ninjalog_uploader.py Loading commit data...
ninjalog_uploader_wrapper.py Loading commit data...
owners.py Loading commit data...
owners_client.py Loading commit data...
owners_finder.py Loading commit data...
pinpoint Loading commit data...
pinpoint.bat Loading commit data...
post_build_ninja_summary.py Loading commit data...
presubmit_canned_checks.py Loading commit data...
presubmit_canned_checks_test.py Loading commit data...
presubmit_canned_checks_test_mocks.py Loading commit data...
presubmit_support.py Loading commit data...
profile.xml Loading commit data...
prpc Loading commit data...
prpc.bat Loading commit data...
pylint Loading commit data...
pylint-1.5 Loading commit data...
pylint-1.5.bat Loading commit data...
pylint-2.6 Loading commit data...
pylint-2.6.bat Loading commit data...
pylint-2.7 Loading commit data...
pylint-2.7.bat Loading commit data...
pylint.bat Loading commit data...
pylint_main.py Loading commit data...
pylintrc Loading commit data...
python_runner.sh Loading commit data...
rdb Loading commit data...
rdb.bat Loading commit data...
rdb_wrapper.py Loading commit data...
repo Loading commit data...
repo_launcher Loading commit data...
roll-dep Loading commit data...
roll-dep.bat Loading commit data...
roll_dep.py Loading commit data...
rustfmt.py Loading commit data...
scm.py Loading commit data...
setup_color.py Loading commit data...
split_cl.py Loading commit data...
subcommand.py Loading commit data...
subprocess2.py Loading commit data...
swift-format Loading commit data...
swift_format.py Loading commit data...
update_depot_tools Loading commit data...
update_depot_tools.bat Loading commit data...
update_depot_tools_toggle.py Loading commit data...
upload_metrics.py Loading commit data...
upload_to_google_storage.py Loading commit data...
utils.py Loading commit data...
vpython Loading commit data...
vpython.bat Loading commit data...
vpython3 Loading commit data...
vpython3.bat Loading commit data...
watchlists.py Loading commit data...
weekly Loading commit data...
whitespace.txt Loading commit data...
win32imports.py Loading commit data...
wtf Loading commit data...
yapf Loading commit data...
yapf.bat Loading commit data...