Skip to content

C
Chromium_Depot_tools
Switch branch/tag
  1. 06 May, 2019 1 commit
  3. 12 Feb, 2018 1 commit
  5. 02 Feb, 2018 1 commit
    • Michael Achenbach's avatar
      Optimize check for existing files in download_from_google_storage · 1f067b88
      Michael Achenbach authored 
      The first call to gsutil causes a network call and makes the download
script slow also in the most optimal cases. This CL refactors the
download script and moves the first gsutil call after checking locally
if sha1s match.

1) This turns the input acquisition into a generator and buffers the
files and sha1s in a list before multithreading.

2) This sequentially checks the sha1s and files and bails out early if
all match. In Chrome-land, we usually call this script with only one
file. There are some cases with around 4. This could also be
parallelized if the need arises.

3) The initial gsutil check, which ensures gsutil is updated, is moved
right in front of the multithreaded downloads.

The performance of one call to download_from_google_storage for an
existing 500MB file is 2.3s before this CL and 1.2s after (most of the
remaining time left is spent for making sha1sum).

Example for full gclient runhooks (when everything is up-to-date):

Chromium: 32s before, 12s after
V8: 12s before, 3s after

Bug: 776311
Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
Reviewed-on: https://chromium-review.googlesource.com/897562
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: 's avatarSergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: 's avatarRyan Tseng <hinoka@chromium.org>
      1f067b88
  7. 03 Nov, 2017 1 commit
  9. 23 Oct, 2017 1 commit
  11. 10 Oct, 2017 1 commit
  13. 09 Oct, 2017 1 commit
  15. 16 Dec, 2016 1 commit
  17. 18 Aug, 2015 1 commit
  19. 29 Jul, 2015 1 commit
  21. 30 Jun, 2015 2 commits
    • boliu@chromium.org's avatar
      Revert of Add support for tar.gz archive files to download from... · 2aa134c4
      boliu@chromium.org authored 
      Revert of Add support for tar.gz archive files to download from download_from_google_storage (patchset #8 id:140001 of https://codereview.chromium.org/807463005/)

Reason for revert:
This conflicts with (and duplicates?) the -z option below and has broken one of the chrome android downstream bots.

Safer to revert than trying to fix

Original issue's description:
> Add support for tar.gz archive files to download from download_from_google_storage
> 
> Also, add support for directly setting the public read flag on uploaded files.
> 
> The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
> 
> I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
> 
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295872

TBR=hinoka@google.com,hinoka@chromium.org,azarchs@chromium.org,ricow@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Review URL: https://codereview.chromium.org/1209033006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295883 0039d316-1c4b-4281-b951-d872f2087c98
      2aa134c4
    • ricow@google.com's avatar
      Add support for tar.gz archive files to download from download_from_google_storage · 0c7d94eb
      ricow@google.com authored 
      Also, add support for directly setting the public read flag on uploaded files.

The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).

I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.

Review URL: https://codereview.chromium.org/807463005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295872 0039d316-1c4b-4281-b951-d872f2087c98
      0c7d94eb
  23. 18 Jun, 2015 1 commit
  25. 17 Jan, 2014 1 commit
    • hinoka@google.com's avatar
      Adds --auto_platform to download_from_google_storage. · c8270632
      hinoka@google.com authored 
      A very common pattern for the users of this script is to have binaries in
three different folders, eg:
some/folder/win/bin.exe
some/folder/mac/bin
some/folder/linux/bin

Instead of using --platform to specify three different paths, we can recognize
this usage and pass in --auto_platform, and then --directory some/folder/ as the
target.

When enumerating files, it will match each individual file to see if they have
(win|mac|linux) as a parent folder name, and filter out non-matching platforms.

BUG=
TEST= src/third_party/clang_format/bin$ download_from_google_storage --auto_platform --no_auth -b chromium-clang-fomat -dr .
0> Downloading ./linux/clang-format...

Review URL: https://codereview.chromium.org/126893002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@245643 0039d316-1c4b-4281-b951-d872f2087c98
      c8270632
  27. 12 Dec, 2013 1 commit
    • hinoka@chromium.org's avatar
      Adds SSO auth to gsutil · c6a2ee69
      hinoka@chromium.org authored 
      Code path:
1. plugins.sso_auth is imported, which adds the AuthHandler class to the global state.
2. HasConfiguredCredentials() in gslib/utils.py is called by gsutil, and will return true if "prodaccess" exists on the system, which tells the system that we don't want a no-op auth handler.
3. When a command is called, all the auth handlers are cycled through and sso_auth.SSOAuth is called, which calls a stubby command to emit a gaiamint'ed oauth2 access token, which is then used as the Authorization Header

if --bypass_prodaccess is passed in, then:
1. HasConfiguredCredentials() will bypass the check for prodaccess, as if it didn't exist.
2. plugins.sso_auth does not get imported.
Which will essentially cause gsutil to behave as if this patch never existed.

So the expected behavior is:
=.boto file does not exist, prodaccess exists, but unauthenticated=
Failure: No handler was ready to authenticate. 3 handlers were checked. ['OAuth2Auth', 'HmacAuthV1Handler', 'SSOAuth'] Check your credentials.

=.boto file exists, prodaccess exists, but unauthenticated=
sso_auth will raise NotReadyToAuthenticate, and the .boto file will be used instead

=.boto file exists, prodaccess exists, authenticated=
sso_auth will be run _after_ the default gsutil authenticator, which causes the sso_auth to be used over whatever the default authentication is.

bypass_prodaccess is passed in by default to upload_to_google_storage because we expect people who use upload_to_google_storage to not need prodaccess and have their own boto file already.  Also the sso_auth plugin will only request a readonlyi token, which will not work for uploading.

BUG=258152

Review URL: https://codereview.chromium.org/86123002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@240266 0039d316-1c4b-4281-b951-d872f2087c98
      c6a2ee69
  29. 25 Nov, 2013 1 commit
  31. 19 Nov, 2013 1 commit
  33. 20 Jun, 2013 1 commit
  35. 04 Jun, 2013 1 commit
  37. 13 Mar, 2013 1 commit