Added gsutil/ to depot_tools/third_party/

This is needed for https://chromiumcodereview.appspot.com/12042069/
Which uses gsutil to download objects from Google Storage based on SHA1 sums

Continuation of: https://chromiumcodereview.appspot.com/12317103/
Rietveld didn't like a giant CL with all of gsutil (kept crashing on upload),
The CL is being split into three parts

Related:
https://chromiumcodereview.appspot.com/12755026/ (gsutil/boto)
https://chromiumcodereview.appspot.com/12685010/ (gsutil/gslib)

BUG=

Review URL: https://codereview.chromium.org/12685009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@188896 0039d316-1c4b-4281-b951-d872f2087c98

third_party/fancy_urllib/README

+The fancy_urllib library was obtained from
+http://googleappengine.googlecode.com/svn/trunk/python/lib/fancy_urllib/fancy_urllib/__init__.py
+under the following license (http://googleappengine.googlecode.com/svn/trunk/python/LICENSE):
+
+
+GOOGLE APP ENGINE SDK
+=====================
+Copyright 2008 Google Inc.
+All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

third_party/gsutil/CHECKSUM

+010822c61d38d70ac23600bc955fccf5

third_party/gsutil/MANIFEST.in

+include gsutil COPYING VERSION LICENSE.third_party README setup.py pkg_util.py
+recursive-include gslib *
+recursive-include oauth2_plugin *
+recursive-include third_party *
+recursive-include boto *

third_party/gsutil/README

+This directory contains the Python command line tool gsutil, which Google
+has released as open source, to demonstrate the Google Storage API and to
+provide a tool for manipulating data in the system.
+
+Prerequisites:
+
+Gsutil requires Python 2.6 or later.
+
+To install gsutil take the following steps:
+
+1. Pick a place where you want to install the software.  You can
+   install the code wherever you prefer; for brevity the instructions below
+   assume you want to install in $HOME/gsutil.
+
+2. To install gsutil on Linux/Unix or MacOS, open a shell window, change
+   directories to where you downloaded the gsutil.tar.gz file, and do this:
+   % tar xfz gsutil.tar.gz -C $HOME
+
+   Then add the following line to your $HOME/.bashrc shell initialization
+   file:
+       export PATH=${PATH}:$HOME/gsutil
+
+   The next time you start a shell you should be able to run gsutil from
+   the command line.
+
+3. To install gsutil on Windows, install cygwin (http://www.cygwin.com/),
+   with at least version 2.6.5 of Python. Once you have that, start a shell
+   and follow the Linux instructions above for unpacking and installing gsutil.
+
+4. The first time you try to run gsutil, it will detect that you have no
+   configuration file containing your credentials, interactively prompt you,
+   and create the file.  
+
+   After this you can use the tool.  Running gsutil with with no arguments
+   will print a help summary.
+
+For more information on installing and using gsutil, see
+<http://code.google.com/apis/storage/docs/gsutil.html>.

third_party/gsutil/README.chromium

+Name: gsutil
+URL: https://github.com/GoogleCloudPlatform/gsutil
+Version: 3.25
+License: Apache 2.0
+
+Description:
+Set of tools to allow querying, uploading, and downloading objects from
+Google Storage.
+
+Modifications:
+* Removed gsutil/gslib/commands/test.py
+* Removed gsutil/pkg_gen.sh
+* Removed gsutil/gslib/tests/
+* Moved gsutil/boto as a depot_tools third_party lib
+* Moved gsutil/third_party into our own third_party directory
+* Append sys.path in gsutil/gsutil to find the moved third_party modules
+* Updated checksum ce71ac982f1148315e7fa65cff2f83e8 -> bf29190007bc7557c33806367ee3ce9e 
+
+Full license is in the COPYING file.

third_party/gsutil/README.pkg

+Package Generation Notes for gsutil
+
+Gsutil can be distributed in one of three ways:
+
+1. legacy mode - User unpacks archive file into a private directory tree 
+and maintains his/her own private copy of gsutil, boto, etc. This is the 
+only supported installation mode for Windows users.
+
+2. enterprise mode - User unpacks the gsutil archive file and runs
+'python setup.py install' (as root), which installs everything into 
+a shared directory tree (/usr/share/gsutil) with a symlink from 
+/usr/bin/gsutil to /usr/share/gsutil/gsutil to provide easy access to
+the shared gsutil command. In enterprise mode, the software gets installed 
+in one shared location, which makes it easier to install, update and 
+manage gsutil for a community of users.
+
+NOTE: Enterprise mode (installing gsutil via setup.py) is no longer 
+officially supported - unpacking the zip file into a directory is the 
+preferred method for installing gsutil for both shared and private 
+configurations.
+
+3. rpm mode - User installs the gsutil rpm package file on a Red Hat
+Linux system using the rpm command. The resulting installation image 
+looks precisely the same as the results of installing with enterprise 
+mode, i.e. a shared directory tree (/usr/share/gsutil) with a symlink 
+from /usr/bin/gsutil. rpm mode is intended for enterprises that want 
+a stable release that does not necessarily contain the latest changes.
+
+All three modes derive their inventory from a common text file called
+MANIFEST.in. If you want to add one or more new files or directories, 
+you only need to edit that one file and all three installation modes
+will automatically inherit the change(s). 
+
+GENERATING PACKAGE FILES
+
+First update the VERSION file and the gsutil.spec files to reflect the 
+new version number.
+
+Legacy mode and enterprise mode are both embodied in the same gsutil
+archive file, the only difference being that the latter entails running 
+one additional command after unpacking the gsutil archive file. So the
+same archive file we've always distributed for gsutil will be used for
+both legacy and enterprise installation modes.
+
+For rpm mode, there's a new tool call pkg_gen.sh, which when run with no
+arguments creates an rpm file at this location:
+
+  $HOME/rpmbuild/RPMS/noarch/gsutil-2.0-1.noarch.rpm
+

third_party/gsutil/VERSION

+3.25

third_party/gsutil/gsutil.spec.in

+#
+# gsutil.spec - RPM specification file for Google Cloud Storage command 
+# line utility (gsutil).
+#
+# Copyright 2011 Google Inc.
+# 
+
+Name: 	   gsutil
+Version:   2.0
+Release:   1%{?dist}
+Summary:   gsutil command line utility for Google Cloud Storage
+License:   ASL 2.0
+Group:     Development/Libraries
+Url:       http://code.google.com/apis/storage/docs/gsutil.html
+Source0:   http://gsutil.googlecode.com/files/%{name}-%{version}.zip
+BuildArch: noarch
+BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
+# Dependency on boto commented out for now because initially we plan to 
+# bundle boto with this package, however, when we're ready to depend on 
+# a separate boto rpm package, this line should be uncommented.
+#Requires:  python-boto
+
+%description
+
+GSUtil is a Python application that facilitates access to Google Cloud Storage 
+from the command line. You can use GSUtil to do a wide range of bucket and 
+object management tasks, including:
+
+- Creating and deleting buckets.
+- Uploading, downloading, and deleting objects.
+- Listing buckets and objects.
+- Moving, copying, and renaming objects.
+- Setting object and bucket ACLs.
+
+%prep
+%setup -q
+
+%build
+python setup.py build
+
+%install
+python setup.py install --skip-build --root=%{buildroot}
+# Make all files and dirs in build area readable by other
+# and make all directories executable by other. These steps
+# are performed in support of the rpm installation mode, 
+# in which users with different user/group than the 
+# installation user/group must be able to run gsutil.
+chmod -R o+r %{buildroot}/usr/share/gsutil
+find %{buildroot}/usr/share/gsutil -type d | xargs chmod o+x
+# Make main gsutil script readable and executable by other.
+chmod o+rx %{buildroot}/usr/share/gsutil/gsutil
+# Remove Python egg file, which we don't use (but setup.py insists on 
+# building) so we remove it here.
+rm %{buildroot}/usr/local/lib/python2.6/dist-packages/gsutil-2.0.egg-info
+# Remove update command, which shouldn't be used when gsutil is managed by RPM.
+rm %{buildroot}/usr/share/gsutil/gslib/commands/update.py
+# Create /usr/bin under buildroot and symlink gsutil so users don't
+# need to add a custom directory to their PATH.
+mkdir -p %{buildroot}%{_bindir}
+cd %{buildroot}%{_bindir}
+ln -s ../share/gsutil/gsutil gsutil
+
+%clean
+rm -rf %{buildroot}
+
+%files
+%defattr(-,root,root,-)
+# Lines ending with a slash cause recursive enumeration of directory contents.
+%{_bindir}/%{name}
+###FILES_GO_HERE###
+
+%changelog
+* Tue Dec 10 2011 Marc Cohen <gs-team@google.com> 2.0-1
+- initial version of rpm spec file for gsutil for inclusion in RHEL
+

third_party/gsutil/oauth2_plugin/__init__.py

+# Copyright 2011 Google Inc. All Rights Reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+"""Package marker file."""

third_party/gsutil/oauth2_plugin/oauth2_helper.py

+# Copyright 2011 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper routines to facilitate use of oauth2_client in gsutil."""
+
+import sys
+import time
+import webbrowser
+
+import oauth2_client
+
+GSUTIL_CLIENT_ID = '909320924072.apps.googleusercontent.com'
+# Google OAuth2 clients always have a secret, even if the client is an installed
+# application/utility such as gsutil.  Of course, in such cases the "secret" is
+# actually publicly known; security depends entirly on the secrecy of refresh
+# tokens, which effectively become bearer tokens.
+GSUTIL_CLIENT_NOTSOSECRET = 'p3RlpR10xMFh9ZXBS/ZNLYUu'
+
+GOOGLE_OAUTH2_PROVIDER_LABEL = 'Google'
+GOOGLE_OAUTH2_PROVIDER_AUTHORIZATION_URI = (
+    'https://accounts.google.com/o/oauth2/auth')
+GOOGLE_OAUTH2_PROVIDER_TOKEN_URI = (
+    'https://accounts.google.com/o/oauth2/token')
+
+OOB_REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
+
+def OAuth2ClientFromBotoConfig(config):
+  token_cache = None
+  token_cache_type = config.get('OAuth2', 'token_cache', 'file_system')
+
+  if token_cache_type == 'file_system':
+    if config.has_option('OAuth2', 'token_cache_path_pattern'):
+      token_cache = oauth2_client.FileSystemTokenCache(
+          path_pattern=config.get('OAuth2', 'token_cache_path_pattern'))
+    else:
+      token_cache = oauth2_client.FileSystemTokenCache()
+  elif token_cache_type == 'in_memory':
+    token_cache = oauth2_client.InMemoryTokenCache()
+  else:
+    raise Exception(
+        "Invalid value for config option OAuth2/token_cache: %s" %
+        token_cache_type)
+
+  proxy = None
+  if (config.has_option('Boto', 'proxy')
+      and config.has_option('Boto', 'proxy_port')):
+    proxy = "%s:%s" % (config.get('Boto', 'proxy'),
+        config.get('Boto', 'proxy_port'))
+
+  provider_label = config.get(
+      'OAuth2', 'provider_label', GOOGLE_OAUTH2_PROVIDER_LABEL)
+  provider_authorization_uri = config.get(
+      'OAuth2', 'provider_authorization_uri',
+      GOOGLE_OAUTH2_PROVIDER_AUTHORIZATION_URI)
+  provider_token_uri = config.get(
+      'OAuth2', 'provider_token_uri', GOOGLE_OAUTH2_PROVIDER_TOKEN_URI)
+
+  client_id = config.get('OAuth2', 'client_id', GSUTIL_CLIENT_ID)
+  client_secret = config.get(
+      'OAuth2', 'client_secret', GSUTIL_CLIENT_NOTSOSECRET)
+
+  return oauth2_client.OAuth2Client(
+      oauth2_client.OAuth2Provider(
+          provider_label, provider_authorization_uri, provider_token_uri),
+      client_id, client_secret,
+      proxy=proxy, access_token_cache=token_cache)
+
+def OAuth2ApprovalFlow(oauth2_client, scopes, launch_browser=False):
+  approval_url = oauth2_client.GetAuthorizationUri(OOB_REDIRECT_URI, scopes)
+  if launch_browser:
+    sys.stdout.write(
+        'Attempting to launch a browser with the OAuth2 approval dialog at '
+        'URL: %s\n\n'
+        '[Note: due to a Python bug, you may see a spurious error message "object is not\n'
+        'callable [...] in [...] Popen.__del__" which can be ignored.]\n\n' % approval_url)
+  else:
+    sys.stdout.write(
+        'Please navigate your browser to the following URL:\n%s\n' %
+        approval_url)
+
+  sys.stdout.write(
+      'In your browser you should see a page that requests you to authorize '
+      'gsutil to access\nGoogle Cloud Storage on your behalf. After you '
+      'approve, an authorization code will be displayed.\n\n')
+  if (launch_browser and
+      not webbrowser.open(approval_url, new=1, autoraise=True)):
+    sys.stdout.write(
+        'Launching browser appears to have failed; please navigate a browser '
+        'to the following URL:\n%s\n' % approval_url)
+  # Short delay; webbrowser.open on linux insists on printing out a message
+  # which we don't want to run into the prompt for the auth code.
+  time.sleep(2)
+  code = raw_input('Enter the authorization code: ')
+
+  refresh_token, access_token = oauth2_client.ExchangeAuthorizationCode(
+      code, OOB_REDIRECT_URI, scopes)
+
+  return refresh_token
+

third_party/gsutil/oauth2_plugin/oauth2_plugin.py

+from boto.auth_handler import AuthHandler
+from boto.auth_handler import NotReadyToAuthenticate 
+import oauth2_client
+import oauth2_helper
+
+class OAuth2Auth(AuthHandler):
+
+  capability = ['google-oauth2', 's3']
+
+  def __init__(self, path, config, provider):
+    if (provider.name == 'google'
+        and config.has_option('Credentials', 'gs_oauth2_refresh_token')):
+
+      self.oauth2_client = oauth2_helper.OAuth2ClientFromBotoConfig(config)
+
+      self.refresh_token = oauth2_client.RefreshToken(
+          self.oauth2_client,
+          config.get('Credentials', 'gs_oauth2_refresh_token'))
+    else:
+      raise NotReadyToAuthenticate()
+
+  def add_auth(self, http_request):
+    http_request.headers['Authorization'] = \
+        self.refresh_token.GetAuthorizationHeader()

third_party/gsutil/pkg_util.py

+#!/usr/bin/env python
+
+# Utilities to facilitate maintaining one master list of package contents 
+# in MANIFEST.in and allow us to import that list into various packaging 
+# tools (e.g. rpmbuid and setup.py).
+
+# Define the file in which we maintain package contents. Rather than 
+# hard-coding our package contents, to ease maintenance we read the 
+# manifest file to obtain the list of files and directories to include.
+MANIFEST_IN = 'MANIFEST.in'
+
+# Define input and output files for customizing the rpm package spec.
+SPEC_IN = 'gsutil.spec.in'
+SPEC_OUT = 'gsutil.spec'
+
+# Root of rpmbuild tree for file enumeration in gsutil.spec file.
+RPM_ROOT = '%{_datadir}/%{name}/'
+
+def parse_manifest(files, dirs):
+  '''Parse contents of manifest file and append results to passed lists
+     of files and directories.
+  '''
+  f = open(MANIFEST_IN, 'r')
+  for line in f:
+    line = line.strip()
+    # Skip empty or comment lines.
+    if (len(line) <= 0) or (line[0] == '#'):
+      continue
+    tokens = line.split()
+    if len(tokens) >= 0:
+      if tokens[0] == 'include':
+        files.extend(tokens[1:])
+      elif tokens[0] == 'recursive-include' and tokens[2] == '*':
+        dirs.append(tokens[1])
+      else:
+        err = 'Unsupported type ' + tokens[0] + ' in ' + MANIFEST_IN + ' file.'
+        raise Exception(err)
+  f.close()
+
+# When executed as a separate script, create a dynamically generated rpm 
+# spec file. Otherwise, when loaded as a module by another script, no
+# specific actions are taken, other than making utility functions available
+# to the loading script.
+if __name__ == '__main__':
+  # Running as main so generate a new rpm spec file.
+  files = []
+  dirs = []
+  parse_manifest(files, dirs)
+  fin = open(SPEC_IN, 'r')
+  fout = open(SPEC_OUT, 'w')
+  for line in fin:
+    if line.strip() == '###FILES_GO_HERE###':
+      for file in files:
+        fout.write(RPM_ROOT + file + '\n')
+      for dir in dirs:
+        fout.write(RPM_ROOT + dir + '/\n')
+    else:
+      fout.write(line)
+  fout.close()
+  fin.close()

third_party/retry_decorator/LICENSE.google

+Copyright (c) 2013, SaltyCrane
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+ * Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the
+   distribution.
+
+ * Neither the name of the SaltyCrane nor the names of its
+   contributors may be used to endorse or promote products derived
+   from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

third_party/retry_decorator/decorators.py

+import time
+from functools import wraps
+
+
+def retry(ExceptionToCheck, tries=4, delay=3, backoff=2, logger=None):
+    """Retry calling the decorated function using an exponential backoff.
+
+    http://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/
+    original from: http://wiki.python.org/moin/PythonDecoratorLibrary#Retry
+
+    :param ExceptionToCheck: the exception to check. may be a tuple of
+        exceptions to check
+    :type ExceptionToCheck: Exception or tuple
+    :param tries: number of times to try (not retry) before giving up
+    :type tries: int
+    :param delay: initial delay between retries in seconds
+    :type delay: int
+    :param backoff: backoff multiplier e.g. value of 2 will double the delay
+        each retry
+    :type backoff: int
+    :param logger: logger to use. If None, print
+    :type logger: logging.Logger instance
+    """
+    def deco_retry(f):
+
+        @wraps(f)
+        def f_retry(*args, **kwargs):
+            mtries, mdelay = tries, delay
+            while mtries > 1:
+                try:
+                    return f(*args, **kwargs)
+                except ExceptionToCheck, e:
+                    msg = "%s, Retrying in %d seconds..." % (str(e), mdelay)
+                    if logger:
+                        logger.warning(msg)
+                    else:
+                        print msg
+                    time.sleep(mdelay)
+                    mtries -= 1
+                    mdelay *= backoff
+            return f(*args, **kwargs)
+
+        return f_retry  # true decorator
+
+    return deco_retry