• Daniel Bratell's avatar
    Speed up some really slow OWNERS operations for large/huge patches · b2b66999
    Daniel Bratell authored
    The current owners systems scales badly on the number of files in a
    patch, on the number of OWNERS files and on the number of "set noparent"
    rules. This patch addresses all three of those.
    
    If you have a patch with many (thousands) of files, you run into a lot
    of repeated work because the parent directories are visited for every
    file. By using a cache or remembering what directories have been
    visited, a lot of work can be avoided.
    
    If you have many OWNERS files or "set noparent" rules in scope of a
    patch, owners and stop_looking lookups became slow.  By splitting the
    rules by longest glob-free directory path, you get much smaller lists
    to process, which makes the code much faster.
    
    (Test case with "all the files" went from >15 minutes to 30 seconds).
    
    This saves about 2.5 second for git cl upload with a patch of
    350 files spread over the tree.
    
    The addition of many noparent rules in the Chromium tree might have
    been the major reason this is needed, but there are also more files to
    change, allowing larger patches.
    
    Bug: 920591
    Change-Id: If21178746ca9b88c2c07d265bd583b556d1734b7
    Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1404172Reviewed-by: 's avatarDirk Pranke <dpranke@chromium.org>
    Auto-Submit: Daniel Bratell <bratell@opera.com>
    Commit-Queue: Daniel Bratell <bratell@opera.com>
    b2b66999
Name
Last commit
Last update
bootstrap/win Loading commit data...
fetch_configs Loading commit data...
git-templates Loading commit data...
infra Loading commit data...
man Loading commit data...
recipes Loading commit data...
support Loading commit data...
testing_support Loading commit data...
tests Loading commit data...
third_party Loading commit data...
win_toolchain Loading commit data...
zsh-goodies Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.style.yapf Loading commit data...
CROS_OWNERS Loading commit data...
LICENSE Loading commit data...
OWNERS Loading commit data...
PRESUBMIT.py Loading commit data...
README.gclient.md Loading commit data...
README.git-cl.md Loading commit data...
README.md Loading commit data...
README.testing Loading commit data...
WATCHLISTS Loading commit data...
annotated_gclient.py Loading commit data...
appengine_mapper.py Loading commit data...
auth.py Loading commit data...
autoninja Loading commit data...
autoninja.bat Loading commit data...
autoninja.py Loading commit data...
bb Loading commit data...
bb.bat Loading commit data...
breakpad.py Loading commit data...
buildbucket.py Loading commit data...
cbuildbot Loading commit data...
checkout.py Loading commit data...
chrome_set_ver Loading commit data...
cipd Loading commit data...
cipd.bat Loading commit data...
cipd.ps1 Loading commit data...
cipd_bin_setup.bat Loading commit data...
cipd_bin_setup.sh Loading commit data...
cipd_client_version Loading commit data...
cipd_client_version.digests Loading commit data...
cipd_manifest.txt Loading commit data...
cipd_manifest.versions Loading commit data...
cit Loading commit data...
cit.bat Loading commit data...
cit.py Loading commit data...
clang-format Loading commit data...
clang-format.bat Loading commit data...
clang_format.py Loading commit data...
clang_format_merge_driver Loading commit data...
clang_format_merge_driver.bat Loading commit data...
clang_format_merge_driver.py Loading commit data...
codereview.settings Loading commit data...
compile_single_file Loading commit data...
compile_single_file.bat Loading commit data...
compile_single_file.py Loading commit data...
cpplint.bat Loading commit data...
cpplint.py Loading commit data...
cpplint_chromium.py Loading commit data...
cros Loading commit data...
cros_sdk Loading commit data...
dart_format.py Loading commit data...
depot-tools-auth Loading commit data...
depot-tools-auth.bat Loading commit data...
depot-tools-auth.py Loading commit data...
detect_host_arch.py Loading commit data...
download_from_google_storage Loading commit data...
download_from_google_storage.bat Loading commit data...
download_from_google_storage.py Loading commit data...
ensure_bootstrap Loading commit data...
fetch Loading commit data...
fetch.bat Loading commit data...
fetch.py Loading commit data...
fix_encoding.py Loading commit data...
gclient Loading commit data...
gclient-new-workdir.py Loading commit data...
gclient.bat Loading commit data...
gclient.py Loading commit data...
gclient_completion.sh Loading commit data...
gclient_eval.py Loading commit data...
gclient_paths.py Loading commit data...
gclient_scm.py Loading commit data...
gclient_utils.py Loading commit data...
gerrit_client.py Loading commit data...
gerrit_util.py Loading commit data...
git-cache Loading commit data...
git-cl Loading commit data...
git-crrev-parse Loading commit data...
git-drover Loading commit data...
git-find-releases Loading commit data...
git-footers Loading commit data...
git-freeze Loading commit data...
git-gs Loading commit data...
git-hyper-blame Loading commit data...
git-map Loading commit data...
git-map-branches Loading commit data...
git-mark-merge-base Loading commit data...
git-nav-downstream Loading commit data...
git-nav-upstream Loading commit data...
git-new-branch Loading commit data...
git-number Loading commit data...
git-rebase-update Loading commit data...
git-rename-branch Loading commit data...
git-reparent-branch Loading commit data...
git-retry Loading commit data...
git-runhooks Loading commit data...
git-squash-branch Loading commit data...
git-thaw Loading commit data...
git-upstream-diff Loading commit data...
git_cache.py Loading commit data...
git_cl.py Loading commit data...
git_cl_completion.sh Loading commit data...
git_common.py Loading commit data...
git_dates.py Loading commit data...
git_drover.py Loading commit data...
git_find_releases.py Loading commit data...
git_footers.py Loading commit data...
git_freezer.py Loading commit data...
git_hyper_blame.py Loading commit data...
git_map.py Loading commit data...
git_map_branches.py Loading commit data...
git_mark_merge_base.py Loading commit data...
git_nav_downstream.py Loading commit data...
git_new_branch.py Loading commit data...
git_number.py Loading commit data...
git_rebase_update.py Loading commit data...
git_rename_branch.py Loading commit data...
git_reparent_branch.py Loading commit data...
git_retry.py Loading commit data...
git_squash_branch.py Loading commit data...
git_upstream_diff.py Loading commit data...
gn Loading commit data...
gn.bat Loading commit data...
gn.py Loading commit data...
gsutil.py Loading commit data...
gsutil.py.bat Loading commit data...
gsutil.vpython Loading commit data...
led Loading commit data...
led.bat Loading commit data...
luci-auth Loading commit data...
luci-auth.bat Loading commit data...
lucicfg Loading commit data...
lucicfg.bat Loading commit data...
mac_toolchain Loading commit data...
metrics.README.md Loading commit data...
metrics.py Loading commit data...
metrics_utils.py Loading commit data...
my_activity.py Loading commit data...
my_reviews.py Loading commit data...
ninja Loading commit data...
ninja-linux32 Loading commit data...
ninja-linux64 Loading commit data...
ninja-mac Loading commit data...
ninja.exe Loading commit data...
ninjalog.README.md Loading commit data...
ninjalog_uploader.py Loading commit data...
ninjalog_uploader_wrapper.py Loading commit data...
owners.py Loading commit data...
owners_finder.py Loading commit data...
patch.py Loading commit data...
post_build_ninja_summary.py Loading commit data...
presubmit_canned_checks.py Loading commit data...
presubmit_support.py Loading commit data...
profile.xml Loading commit data...
prpc Loading commit data...
prpc.bat Loading commit data...
pylint Loading commit data...
pylint.py Loading commit data...
pylintrc Loading commit data...
python_runner.sh Loading commit data...
repo Loading commit data...
rietveld.py Loading commit data...
roll-dep Loading commit data...
roll-dep-svn Loading commit data...
roll-dep-svn.bat Loading commit data...
roll-dep.bat Loading commit data...
roll_dep.py Loading commit data...
roll_dep_svn.py Loading commit data...
scm.py Loading commit data...
setup_color.py Loading commit data...
split_cl.py Loading commit data...
subcommand.py Loading commit data...
subprocess2.py Loading commit data...
update_depot_tools Loading commit data...
update_depot_tools.bat Loading commit data...
update_depot_tools_toggle.py Loading commit data...
upload_metrics.py Loading commit data...
upload_to_google_storage.py Loading commit data...
vpython Loading commit data...
vpython.bat Loading commit data...
watchlists.py Loading commit data...
weekly Loading commit data...
wtf Loading commit data...
yapf Loading commit data...
yapf.bat Loading commit data...