• Michael Achenbach's avatar
    Optimize check for existing files in download_from_google_storage · 1f067b88
    Michael Achenbach authored
    The first call to gsutil causes a network call and makes the download
    script slow also in the most optimal cases. This CL refactors the
    download script and moves the first gsutil call after checking locally
    if sha1s match.
    
    1) This turns the input acquisition into a generator and buffers the
    files and sha1s in a list before multithreading.
    
    2) This sequentially checks the sha1s and files and bails out early if
    all match. In Chrome-land, we usually call this script with only one
    file. There are some cases with around 4. This could also be
    parallelized if the need arises.
    
    3) The initial gsutil check, which ensures gsutil is updated, is moved
    right in front of the multithreaded downloads.
    
    The performance of one call to download_from_google_storage for an
    existing 500MB file is 2.3s before this CL and 1.2s after (most of the
    remaining time left is spent for making sha1sum).
    
    Example for full gclient runhooks (when everything is up-to-date):
    
    Chromium: 32s before, 12s after
    V8: 12s before, 3s after
    
    Bug: 776311
    Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
    Reviewed-on: https://chromium-review.googlesource.com/897562
    Commit-Queue: Michael Achenbach <machenbach@chromium.org>
    Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
    Reviewed-by: 's avatarRyan Tseng <hinoka@chromium.org>
    1f067b88
Name
Last commit
Last update
bootstrap/win Loading commit data...
fetch_configs Loading commit data...
git-templates Loading commit data...
infra Loading commit data...
man Loading commit data...
recipes Loading commit data...
support Loading commit data...
testing_support Loading commit data...
tests Loading commit data...
third_party Loading commit data...
win_toolchain Loading commit data...
zsh-goodies Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.style.yapf Loading commit data...
LICENSE Loading commit data...
OWNERS Loading commit data...
PRESUBMIT.py Loading commit data...
README.gclient.md Loading commit data...
README.git-cl.md Loading commit data...
README.md Loading commit data...
README.testing Loading commit data...
WATCHLISTS Loading commit data...
annotated_gclient.py Loading commit data...
appengine_mapper.py Loading commit data...
apply_issue Loading commit data...
apply_issue.bat Loading commit data...
apply_issue.py Loading commit data...
auth.py Loading commit data...
autoninja Loading commit data...
autoninja.bat Loading commit data...
autoninja.py Loading commit data...
breakpad.py Loading commit data...
buildbucket.py Loading commit data...
cbuildbot Loading commit data...
checkout.py Loading commit data...
chrome_set_ver Loading commit data...
cipd Loading commit data...
cipd.bat Loading commit data...
cipd.ps1 Loading commit data...
cipd_bin_setup.bat Loading commit data...
cipd_bin_setup.sh Loading commit data...
cipd_client_version Loading commit data...
cipd_manifest.txt Loading commit data...
cit Loading commit data...
cit.bat Loading commit data...
cit.py Loading commit data...
clang-format Loading commit data...
clang-format.bat Loading commit data...
clang_format.py Loading commit data...
clang_format_merge_driver Loading commit data...
clang_format_merge_driver.bat Loading commit data...
clang_format_merge_driver.py Loading commit data...
codereview.settings Loading commit data...
compile_single_file Loading commit data...
compile_single_file.bat Loading commit data...
compile_single_file.py Loading commit data...
cpplint.bat Loading commit data...
cpplint.py Loading commit data...
cpplint_chromium.py Loading commit data...
cros Loading commit data...
cros_sdk Loading commit data...
dart_format.py Loading commit data...
depot-tools-auth Loading commit data...
depot-tools-auth.bat Loading commit data...
depot-tools-auth.py Loading commit data...
download_from_google_storage Loading commit data...
download_from_google_storage.bat Loading commit data...
download_from_google_storage.py Loading commit data...
ensure_bootstrap Loading commit data...
fetch Loading commit data...
fetch.bat Loading commit data...
fetch.py Loading commit data...
fix_encoding.py Loading commit data...
gclient Loading commit data...
gclient-new-workdir.py Loading commit data...
gclient.bat Loading commit data...
gclient.py Loading commit data...
gclient_completion.sh Loading commit data...
gclient_eval.py Loading commit data...
gclient_scm.py Loading commit data...
gclient_utils.py Loading commit data...
gerrit_client.py Loading commit data...
gerrit_util.py Loading commit data...
git-cache Loading commit data...
git-cl Loading commit data...
git-crrev-parse Loading commit data...
git-crsync Loading commit data...
git-drover Loading commit data...
git-find-releases Loading commit data...
git-footers Loading commit data...
git-freeze Loading commit data...
git-gs Loading commit data...
git-hyper-blame Loading commit data...
git-map Loading commit data...
git-map-branches Loading commit data...
git-mark-merge-base Loading commit data...
git-nav-downstream Loading commit data...
git-nav-upstream Loading commit data...
git-new-branch Loading commit data...
git-number Loading commit data...
git-rebase-update Loading commit data...
git-rename-branch Loading commit data...
git-reparent-branch Loading commit data...
git-retry Loading commit data...
git-runhooks Loading commit data...
git-squash-branch Loading commit data...
git-thaw Loading commit data...
git-upstream-diff Loading commit data...
git_cache.py Loading commit data...
git_cl.py Loading commit data...
git_cl_completion.sh Loading commit data...
git_common.py Loading commit data...
git_dates.py Loading commit data...
git_drover.py Loading commit data...
git_find_releases.py Loading commit data...
git_footers.py Loading commit data...
git_freezer.py Loading commit data...
git_hyper_blame.py Loading commit data...
git_map.py Loading commit data...
git_map_branches.py Loading commit data...
git_mark_merge_base.py Loading commit data...
git_nav_downstream.py Loading commit data...
git_new_branch.py Loading commit data...
git_number.py Loading commit data...
git_rebase_update.py Loading commit data...
git_rename_branch.py Loading commit data...
git_reparent_branch.py Loading commit data...
git_retry.py Loading commit data...
git_squash_branch.py Loading commit data...
git_upstream_diff.py Loading commit data...
gn Loading commit data...
gn.bat Loading commit data...
gn.py Loading commit data...
gsutil.py Loading commit data...
gsutil.vpython Loading commit data...
led Loading commit data...
led.bat Loading commit data...
mac_toolchain Loading commit data...
my_activity.py Loading commit data...
my_reviews.py Loading commit data...
ninja Loading commit data...
ninja-linux32 Loading commit data...
ninja-linux64 Loading commit data...
ninja-mac Loading commit data...
ninja.exe Loading commit data...
owners.py Loading commit data...
owners_finder.py Loading commit data...
patch.py Loading commit data...
presubmit_canned_checks.py Loading commit data...
presubmit_support.py Loading commit data...
profile.xml Loading commit data...
pylint Loading commit data...
pylint.py Loading commit data...
pylintrc Loading commit data...
python_runner.sh Loading commit data...
repo Loading commit data...
rietveld.py Loading commit data...
roll-dep Loading commit data...
roll-dep-svn Loading commit data...
roll-dep-svn.bat Loading commit data...
roll-dep.bat Loading commit data...
roll_dep.py Loading commit data...
roll_dep_svn.py Loading commit data...
scm.py Loading commit data...
setup_color.py Loading commit data...
split_cl.py Loading commit data...
subcommand.py Loading commit data...
subprocess2.py Loading commit data...
update_depot_tools Loading commit data...
update_depot_tools.bat Loading commit data...
update_depot_tools_toggle.py Loading commit data...
upload_to_google_storage.py Loading commit data...
vpython Loading commit data...
vpython.bat Loading commit data...
watchlists.py Loading commit data...
weekly Loading commit data...
wtf Loading commit data...