Tested manually since we have no unit tests that cover this bit.

R=gavinmak@google.com

Bug: 1105747
Change-Id: I113e27236ed450490e59df9a6f698925ff556bc7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685350Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

When PRESUBMIT is run with a custom --upstream, dry run cl format should
also use this value, otherwise format warnings would be inconsistent.

Change-Id: I1370f614618ecc03706e54cd4b151a207d9b3bae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688631
Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: b/234078033
Change-Id: I54589891b54986859db539e45c3a208f97081bc6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3689029Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>

Currently, the ninja binaries in depot_tools are committed to the repo directly.
This CL installs CIPD ninja packages to replace them.
Switching to CIPD ninja will be done in the next CL.

Also, see the performance comparison and CIPD packages in the comments below.
https://crbug.com/931218#c84
https://crbug.com/931218#c86

Bug: 931218
Change-Id: I44d5e9ea1447588b32a10db20cfba088dd748f48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674981
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>

Bug:1330629
Change-Id: I522255528e36d4806eada70c22afee1277a6778d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668429Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>

This CL makes fetch default to sso:// within the cog env. It also adds a helper within gclient_scm to check if the env is cog.

Bug: chrome-operations:170
Change-Id: I8c35348653406549fadbd08df3d9991bb24ca776
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3670721Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Remove unnecessary os.listdir call and fix infinite loop on Windows if
no file was found in a tree.

Change-Id: I82a8763e807bbc0ce6fcae6b35a370ffe3b34943
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3687234Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>

This reverts commit 013e47be.

Reason for revert: This will be relanded as a warning, although the
bugs which the original change exposed should be fixed soon.

Original change's description:
> Revert "Fix not-run-test detection."
>
> This reverts commit 41691abe.
>
> Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.
>
> Original change's description:
> > Fix not-run-test detection.
> >
> > In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> > the problem of some tests being run neither on Python 2 or Python 3. At
> > the same time reporting was added to warn if tests were run on neither
> > Python version.
> >
> > Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> > object was created but was not added to the list of results, so it had
> > no effect. Also the skipped tests name was added where a list was
> > expected which meant that each character of the test name showed up on
> > its own line.
> >
> > This change fixes the reporting. It also changes the format of the
> > report and makes it an error - a failure to run tests at all deserves
> > that severity level.
> >
> > A test run with the fixed check showed that no errors have crept in -
> > all tests were being run with one or the other. However the fix did
> > find a bug in an in-progress change I was working on.
> >
> > Bug: 1223478
> > Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> > Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> > Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> > Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
>
> Bug: 1223478
> Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

Bug: 1223478, 1330859
Change-Id: I6a0a78e70db77c9c5fda5bab27960cd4e71279e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685349Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

This reverts commit c03ebf0e.

Reason for revert: the original change was not the cause of the test
failures. The actual cause was crrev.com/c/3683105 which was reverted.

Original change's description:
> Revert "Add Python 3 support to GetPythonUnitTests"
>
> This reverts commit dfc71bbe.
>
> Reason for revert: Breaks all presubmits that do not have skip_shebang_check=True set.
>
> Original change's description:
> > Add Python 3 support to GetPythonUnitTests
> >
> > GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
> > scripts using Python 2. This change adds a python3=False parameter to
> > GetPythonUnitTests to give the option of switching to Python 3, while
> > leaving the default as Python 2. The child scripts are run under one or
> > the other, based on this parameter.
> >
> > This change is needed in support of crrev.com/c/3679801.
> >
> > Bug: 1313804
> > Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
> > Reviewed-by: Jesse McKenna <jessemckenna@google.com>
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
>
> Bug: 1313804
> Change-Id: I1bd7096b6cfdfed6205e6dc0b5d4498b5b821b97
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Christoffer Jansson <jansson@chromium.org>
> Owners-Override: Ben Pastene <bpastene@chromium.org>
> Reviewed-by: Tomas Gunnarsson <tommi@chromium.org>

Bug: 1313804
Change-Id: I15ed90cc35d88ece71eb681acc0acc06cb5c3bcc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685229
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Jesse McKenna <jessemckenna@google.com>

This is a reland of commit 6817010e

Additional Changes:
This reland fixes https://crbug.com/1330995 by stopping gclient from accidentally updating the scheme when one is not present.

Original change's description:
> Update fetch protocol using --protocol-override flag
>
> This CL updates gclient sync to use the protocol of the URL specified in the solutions for cloning all the child dependencies of it.
>
> Bug: chrome-operations:170
> Change-Id: I33588059788b677fbae8c3b434100af5c7979a67
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3631600
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1330995
Change-Id: I1447a5e884e41d671d8556c35193f1635f2f6936
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3684112Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Currently cipd packages listed in src/DEPS are being created and
installed in symlink mode (see bug for more info). This CL adds `$OverrideInstallMode copy` to the ensure files for cipd packages installed during gclient runhooks. `$OverrideInstallMode` was
implemented here, https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/3124202,
so that every cipd package can be installed directly to the installation
target, instead of being symlinked to an executable inside the cache/builder/.cipd dir.

The added `$OverrideInstallMode copy` can be removed once all cipd
packages themselves or the default mode is updated to be created in
copy mode.

Bug: 1329641
Change-Id: I68dc718e20f193c384d005c9b6b87603753943d4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673698Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>

This reverts commit 6817010e.

Reason for revert: Breaks gclient sync for flutter

Original change's description:
> Update fetch protocol using --protocol-override flag
>
> This CL updates gclient sync to use the protocol of the URL specified in the solutions for cloning all the child dependencies of it.
>
> Bug: chrome-operations:170
> Change-Id: I33588059788b677fbae8c3b434100af5c7979a67
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3631600
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: chrome-operations:170
Change-Id: I76a49030f48fa266eeea336307e7153b22ded4e9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3684110
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Should be landed once the new cipd is rolled out everywhere (including
bots)

Bug: 1329641
Change-Id: I32edaa167dd26dbf891435215eeb7c833aba091f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680100Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>

This reverts commit 41691abe.

Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.

Original change's description:
> Fix not-run-test detection.
>
> In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> the problem of some tests being run neither on Python 2 or Python 3. At
> the same time reporting was added to warn if tests were run on neither
> Python version.
>
> Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> object was created but was not added to the list of results, so it had
> no effect. Also the skipped tests name was added where a list was
> expected which meant that each character of the test name showed up on
> its own line.
>
> This change fixes the reporting. It also changes the format of the
> report and makes it an error - a failure to run tests at all deserves
> that severity level.
>
> A test run with the fixed check showed that no errors have crept in -
> all tests were being run with one or the other. However the fix did
> find a bug in an in-progress change I was working on.
>
> Bug: 1223478
> Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> Reviewed-by: Fabrice de Gans <fdegans@chromium.org>

Bug: 1223478
Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

This CL updates gclient sync to use the protocol of the URL specified in the solutions for cloning all the child dependencies of it.

Bug: chrome-operations:170
Change-Id: I33588059788b677fbae8c3b434100af5c7979a67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3631600Reviewed-by: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

This reverts commit dfc71bbe.

Reason for revert: Breaks all presubmits that do not have skip_shebang_check=True set.

Original change's description:
> Add Python 3 support to GetPythonUnitTests
>
> GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
> scripts using Python 2. This change adds a python3=False parameter to
> GetPythonUnitTests to give the option of switching to Python 3, while
> leaving the default as Python 2. The child scripts are run under one or
> the other, based on this parameter.
>
> This change is needed in support of crrev.com/c/3679801.
>
> Bug: 1313804
> Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
> Reviewed-by: Jesse McKenna <jessemckenna@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Bug: 1313804
Change-Id: I1bd7096b6cfdfed6205e6dc0b5d4498b5b821b97
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Christoffer Jansson <jansson@chromium.org>
Owners-Override: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Tomas Gunnarsson <tommi@chromium.org>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8812608469227296113

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/5ba6e0d0cd472c2e7783bc95424b44bb46ce2749
  5ba6e0d (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from 27d906266697 to 7bf7ff13b591

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I7ee0f0de3825a12d2c478bedfb372d6cadf38a4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3681981
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8812613803095882881

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/d14718e984d2b7b539f8f9c8bec3215807c0a681
  d14718e (alexschulze@chromium.org)
      [file] Add fail_silently to copy and copytree

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I10eff17643f710878340d5da77d4ebf6e6813ac2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3679516
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
scripts using Python 2. This change adds a python3=False parameter to
GetPythonUnitTests to give the option of switching to Python 3, while
leaving the default as Python 2. The child scripts are run under one or
the other, based on this parameter.

This change is needed in support of crrev.com/c/3679801.

Bug: 1313804
Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099Reviewed-by: Jesse McKenna <jessemckenna@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

In crrev.com/c/2986400 the skip_shebang_check flag was added to address
the problem of some tests being run neither on Python 2 or Python 3. At
the same time reporting was added to warn if tests were run on neither
Python version.

Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
object was created but was not added to the list of results, so it had
no effect. Also the skipped tests name was added where a list was
expected which meant that each character of the test name showed up on
its own line.

This change fixes the reporting. It also changes the format of the
report and makes it an error - a failure to run tests at all deserves
that severity level.

A test run with the fixed check showed that no errors have crept in -
all tests were being run with one or the other. However the fix did
find a bug in an in-progress change I was working on.

Bug: 1223478
Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
Reviewed-by: Fabrice de Gans <fdegans@chromium.org>

Some presubmit failures result in uncaught exceptions which historically
have halted presubmit runs. This behavior makes it more difficult to
find all failures, measure presubmit times, or detect other behavior
over time. This is particularly problematic when running "git cl
presubmit --all".

As an example, presubmit --all was broken and would not complete from
when crrev.com/3642633 landed until crrev.com/c/3657600. The minimal
repro was this presubmit command:

    git cl presubmit --files ui\accessibility\ax_mode.h

This command, or presubmit --all, triggered this error, which halts the
presubmits:

Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'
Traceback (most recent call last):
  File "presubmit_support.py", line 2038, in <module>
    sys.exit(main())
  File "presubmit_support.py", line 2015, in main
    return DoPresubmitChecks(
  File "presubmit_support.py", line 1738, in DoPresubmitChecks
    results += executer.ExecPresubmitScript(presubmit_script, filename)
  File "presubmit_support.py", line 1602, in ExecPresubmitScript
    self._run_check_function(function_name, context, sink,
  File "presubmit_support.py", line 1640, in _run_check_function
    six.reraise(e_type, e_value, e_tb)
  File "C:\Users\brucedawson\.vpython-root\e726d2\lib\site-packages\six.py", line 686, in reraise
    raise value
  File "presubmit_support.py", line 1630, in _run_check_function
    result = eval(function_name + '(*__args)', context)
  File "<string>", line 1, in <module>
  File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
    errs.extend(CheckModesMatch(input_api, output_api))
  File "PRESUBMIT.py", line 283, in CheckModesMatch
    ax_modes_in_js = GetAccessibilityModesFromFile(
  File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
    for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'

With this change the exception is handled and presubmits continue after
printing this message:

Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js', Traceback (most recent call last):
  File "presubmit_support.py", line 1630, in _run_check_function
    result = eval(function_name + '(*__args)', context)
  File "<string>", line 1, in <module>
  File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
    errs.extend(CheckModesMatch(input_api, output_api))
  File "PRESUBMIT.py", line 283, in CheckModesMatch
    ax_modes_in_js = GetAccessibilityModesFromFile(
  File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
    for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'

That is, the crucial information regarding the failure is printed, but
subsequent presubmits are not skipped.

Bug: 1309977
Change-Id: I7cfeda85c830e6c8e567c0df3c50f27af1dbe835
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3653978Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8812990573718281121

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/e52acbc0987bdb31369aba51894b98286a418582
  e52acbc (andrewlamb@chromium.org)
      [recipes-py] roll recipe proto.

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I1a16c436667dfcb2a44c760e2d59ad30cf8d8a7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673905
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

Bug:1328653
Change-Id: I812eae65b6ea4f5bcd01d315405e90e0b3f1e732
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673717
Auto-Submit: Ryan Heise <heiserya@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

R=gavinmak@google.com

Change-Id: I7d9ca9530640ffa4b8e60c247f809b0ff477d88e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673716
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>

A typical cpplint error report during a presubmit looks like this:

    .../linux_ui.cc:64:  Almost always, snprintf is better than strcpy  [runtime/printf] [4]
    ...
    Changelist failed cpplint.py check.

The problem is that the details of the error go to stderr when
CheckChangeLintsClean is run, whereas the "Changelist failed cpplint.py
check." message is printed at the end of the run. If there a lot of
errors, warnings, or messages then the association between them can be
completely lost.

This change adds the tag (cpplint) to the error messages, and adds a
comment to the "failed cpplint.py check" message to suggest looking for
the tag.

Bug: 1309977
Change-Id: I0d073b57f215e28495cbc3e009cab6ac2f23b65e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3632947
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>

This is to fix
https://ci.chromium.org/ui/p/chromium/builders/ci/Win%20x64%20Builder%20(py2%20less)/1151/overview

Bug: 1327534
Change-Id: I3dcead03d0f4e213e1c4e610d8b90fe95ea843d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3670999
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Bug: b/233146956
Change-Id: I15ff4a79e4edf05f26bbb1cd3f7abfc99f209fc8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668430Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>

Some of the expensive checks when running presubmit --all, such as
CheckStableMojomChanges (~300 s) and CheckAddedDepsHaveTargetApprovals
(~200 s) only look at diffs and are therefore guaranteed to be NOPs when
running presubmit --all or --files=. Passing along the no_diffs state
lets these expensive checks be skipped, thus allowing for faster
iteration times.

Initial testing suggests that (with some supporting changes in the
Chromium repo) this reduces "presubmit --all" times by about ten
minutes, or a bit more than 10%, and additional improvements may be
possible.

Special handling for the no-diffs case also offers a simple way to avoid
presubmit failures that happen whenever all files are flagged as being
changed.

Finally, and perhaps most importantly for having a presubmit --all bot,
when --no_diffs is passed we can treat errors like "Issue wasn't
uploaded" and "Add a description to the CL" as messages, thus making it
possible to have zero presubmit errors when run on origin/main.

Bug: 1320937, 1322936
Change-Id: I0d09dd4aae8fdaa48c8b2f89337441cf96dcff72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3628368
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813160854717135793

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/d7417d09a5a31edda68c3d086752704f39e56323
  d7417d0 (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from c73a8d4e4fa5 to 7d578d09794d

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I0fe76bd2c53d93f19e37959baaa024b5ec19279f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3669677
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

Presubmit message types were printed in an inconsistent order, generally
determined by which type was encountered first. This unpredictability
can be confusing when comparing runs. This change enforces a consistent
order, in increasing order of severity. ERRORS always go last so that
they will be the most visible when running presubmits locally.

This also adds a "There were Python %d presubmit errors." message to
the end to make it easier to tell when there were presubmit errors.
This is particularly useful when manually running presubmits.

Bug: 1309977
Change-Id: Ib2b73c7625789bad5b21ae12abf238b746cd11e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668724Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

In https://crbug.com/1327371, we started encountering hanging gsutil
calls. These were triggered under very specific circumstances, but
were essentially due to a multi-processing bug in python on MacOS:
https://bugs.python.org/issue33725

When running gsutil on mac, it explicitly suggests disabling
multi-processing for this reason:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331

So this CL simply puts that suggestion into practice. There are two
options for doing so: adding a line to an active Boto file, or adding
a "-o" arg to every gsutil invocation. Since chromium has multiple
copies of gsutil and multiple different locations where `gsutil` is
invoked, the latter option is considered intractable.

Instead, we choose the Boto file option by adding a context manager
to the gsutil recipe module that will insert a custom Boto file into
the environment that only disables multi-threading. Due to the fact
that the BOTO_PATH env var takes multiple paths, we can safely
incorporate both our custom Boto along with LUCI's per-build Boto file
in the same gsutil invocation. And when the context manager exits, the
environment should revert back to its original state.

Chromium incorporates this method in crrev.com/c/3662023. See it take
effect during the "gclient runhooks" step in this led build:
https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com

Bug: 1327371
Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813260264718091857

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/edc3cd1e9d177fd0ae245fbb9d25e141191add5f
  edc3cd1 (nicholasbishop@google.com)
      walkthrough.md: update test specification style

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ifa6fd468598e783fcbd1da3c4d1165d77823f126
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3664180
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

It's possible that remote closes connection, which would result in a
stacktrace being shown in the stderr. Silence this exception, as we do
with other exceptions.

R=gavinmak@google.com

Bug: 1328518
Change-Id: Ifae2aca64fe5d572c0b71008f6dacbb63ba77e06
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3664178Reviewed-by: Gavin Mak <gavinmak@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813328211131657745

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/80a446c361907cadba554956fa30275ad03341ac
  80a446c (chromium-autoroll@skia-public.iam.gserviceaccount.com)
      Roll CAS Client from e6630cac546f to a80cc32b1739

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: Id68de30c0a778fb53781ac3e53ade56ebf63b5fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661518
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813329470605599937

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1caa943e0c00281cc2901a63b18fa34a988c0825
  1caa943 (bpastene@chromium.org)
      context: Correct misleading comment about inspect the recipe env

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I509e3c5f308fd7ebf127d78985047a430348f823
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661516
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813337018642090161

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/ebae866e0d063e60924ab575d39b5c1e222bfe0d
  ebae866 (mohrr@google.com)
      [url] Correct default step name in docstrings

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ic2f9e599f439ff9b0ca91314df474b01572112b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661511
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

While enabling cpplint on base I found an extraneous error and an overly
complicated invocation of CheckChangeLintsClean. This change disables
the extraneous error and improves the file filtering defaults to make it
easier to invoke cpplint.

The warning was:

    base\win\win_util.cc(193):  Missing username in TODO; it should look
    like "// TODO(my_username): Stuff."  [readability/todo] [2]

Chromium has over 10,000 TODO comments that are links to bugs instead of
users so we clearly want to suppress this warning.

Most calls to CheckChangeLintsClean in Chromium pass in custom file
filtering in order to only analyze C++ files. Failing to pass a file
filter leads to errors like this:

    Ignoring base\win\embedded_i18n\create_string_rc.py; not a valid
    file name (cc, cuh, cu, cpp, h)
    Ignoring c:\src\chromium\src\components\arc\PRESUBMIT.py; not a
    valid file name (cu, cc, cuh, cpp, h)

It doesn't make sense for the canned checks version of
CheckChangeLintsClean to default to passing in files that cpplint will
complain about so if source_file_filter is None then I configure an
appropriate filter.

This change also switches to vs7 output format by default on Windows,
and deletes a trailing space.

Bug: 1309977
Change-Id: I342ca9e56a41502755dfc09307b441dfa6a48ec0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3655592Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813415663324560977

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/ac1405bf5b5ab66894d7a81429c7506a2605a5aa
  ac1405b (atyfto@google.com)
      [cipd] Add ensure_file_resolve() function

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I3bae2f77b2d8bb469fe5926101cedc9e20ef8234
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3659577
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.

The build that created this CL was
https://ci.chromium.org/b/8813625169557805905

Please review the expectation changes, and LGTM+CQ.

recipe_engine:
https://chromium.googlesource.com/infra/luci/recipes-py.git/+/a58dd35fb76ad00a77bd44508f1b6aaafdbb1a2b
  a58dd35 (tikuta@chromium.org)
      file: use vpython3 to run scripts in resources dir

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=iannucci@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I116fa1e5faafade1c85a4cca7077b926af788a8d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3657049
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>

This is lifted from the cpplint 1.6 release at
https://github.com/cpplint/cpplint, with minor
modifications to match the existing docstring style.

Change-Id: I20aff08805e044883717001856d99942690b45c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3648533Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: David Sanders <dsanders11@ucsbalum.com>