[win_tools] Use bundled Python CIPD packages.

Enable bundled Python CIPD packages in bleeding-edge mode. This
replaces the ZIP unpacking approach used before, and introduces
validation and management through the CIPD tool. The bleeding edge
version will only install if a sentinel file is present in the
"depot_tools" root; otherwise, default behavior will continue.

This method adds a upgrade and downgrade path to/from ZIP and
CIPD installations. This is done by rewriting the "win_tools.bat"
process:

1) Ensure that a bootstrap Python is present.
2) Use it to run "win_tools.py", which has the functionality of
    "git_bootstrap.py" plus Python installation.
3) Run "win_tools.py" with appropriate flags.

Some tricks were employed to handle cases where there is an
already-running Python instance that uses the current Python
installation and executable. This happens on bots because the
system uses the same "depot_tools" checkout at multiple launch
layers. To this end, we use the "python.bat" as the "current Python"
authority and refrain from cleaning up old Python directories if their
"python.exe" binaries are currently in use.

We change the Git bleeding edge file to share the same
sentinel file as Python, ".bleeding_edge".

The new Python should have the same facilities as the original Python
bundle.

BUG=chromium:740171
TEST=local

Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736
Reviewed-on: https://chromium-review.googlesource.com/563036
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

.gitignore

@@ -14,6 +14,7 @@
 /git.bat
 /gitk.bat
 /pylint.bat
+/.bleeding_edge
 /.pylint.d/
 /python
 /python.bat
@@ -22,7 +23,7 @@
 /svn
 /svn.bat
 /svnversion.bat
-/.bleeding_edge
+/python_bin_reldir.txt
 /.codereview_upload_cookies
 /.gitconfig
 /.git_bleeding_edge
@@ -33,6 +34,7 @@
 
 # Ignore locations where third-party tools are placed during bootstrapping.
 /python*_bin
+/win_tools*_bin
 /git_bin
 /git-*_bin
 /svn_bin

bootstrap/win/README.md

@@ -3,6 +3,14 @@
 This directory has the 'magic' for the `depot_tools` windows binary update
 mechanisms.
 
+A previous Python may actually be in use when it is run, preventing us
+from replacing it outright without breaking running code. To
+ommodate this, and Python cleanup, we handle Python in two stages:
+
+1. Use CIPD to install both Git and Python at once.
+2. Use "win_tools.py" as a post-processor to install generated files and
+   fix-ups.
+
 ## Software bootstrapped
   * Python (https://www.python.org/)
   * Git for Windows (https://git-for-windows.github.io/)
@@ -18,31 +26,28 @@ work.
 package is present, and if so, if it's the expected version. If either of those
 cases is not true, it will download and unpack the respective binary.
 
-Downloading is done with [get_file.js](./get_file.js), which is a windows script
-host javascript utility to vaguely impersonate `wget`.
+Installation of Git and Python is done by the [win_tools.bat](./win_tools.bat)
+script, which uses CIPD (via the [cipd](/cipd.bat) bootstrap) to acquire and
+install each package into the root of the `depot_tools` repository. Afterwards,
+the [win_tools.py](./win_tools.py) Python script is invoked to install stubs,
+wrappers, and support scripts into `depot_tools` for end-users.
 
-Through a comedy of history, each binary is stored and retrieved differently.
+### Manifest
 
-### Git
+The Git and Python versions are specified in [manifest.txt](./manifest.txt).
 
-Git installs are mirrored versions of the official Git-for-Windows Portable
-releases.
-  * Original: `https://github.com/git-for-windows/git/releases`
-  * Mirror: `gs://chrome-infra/PortableGit*.7z.exe`
+There is an associated file,
+[manifest_bleeding_edge.txt](./manifest_bleeding_edge.txt), that can be used
+to canary new versions on select bots. Any bots with a `.bleeding_edge` file
+in their `depot_tools` root will automatically use the bleeding edge manifest.
+This allows opt-in systems to test against new versions of Python or Git. Once
+those versions have been verified correct, `manifest.txt` can be updated to the
+same specification, which will cause the remainder of systems to update.
 
-#### Updating git version
-  1. Download the new `PortableGit-X.Y.Z-{32,64}.7z.exe` from the
-     git-for-windows release page.
-  1. From either console.developers.google.com or the CLI, do:
-    1. Upload those to the gs://chrome-infra Google Storage bucket.
-    1. Set the `allUsers Reader` permission (click the "Public link" checkbox
-       next to the binaries).
-  1. Edit the `git_version.txt` or `git_version_bleeding_edge.txt` file to
-     be the new version.
-    1. You can use the bleeding edge version to get early feedback/stage a
-       rollout/etc. Users can select this version by 'touch'ing the
-       `.git_bleeding_edge` file in the root depot_tools directory.
-  1. Commit the CL.
+### Bundles
+
+Git and Python bundle construction is documented in
+[infra packaging](https://chromium.googlesource.com/infra/infra/+/master/doc/packaging/).
 
 Note that in order for the update to take effect, `gclient` currently needs to
 run twice. The first time it will update the `depot_tools` repo, and the second
@@ -50,9 +55,79 @@ time it will see the new git version and update to it. This is a bug that should
 be fixed, in case you're reading this and this paragraph infuriates you more
 than the rest of this README.
 
-### Python
+## Testing
+
+After any modification to this script set, a test sequence should be run on a
+Windows bot.
+
+The post-processing will regenerate "python.bat" to point to the current
+Python instance. Any previous Python installations will stick around, but
+new invocations will use the new instance. Old installations will die
+off either due to processes terminating or systems restarting. When this
+happens, they will be cleaned up by the post-processing script.
+
+Testing
+=======
+
+For each of the following test scenarios, run these commands and verify that
+they are working:
+
+```bash
+# Assert that `gclient` invocation will update (and do the update).
+gclient version
+
+# Assert that Python fundamentally works.
+python -c "import psutil; dir(psutil)"
+
+# Assert that Python scripts work from `cmd.exe`.
+git map-branches
+
+# Assert that `git bash` works.
+git bash
+
+## (Within `git bash`) assert that Python fundamentally works.
+python -c "import psutil; dir(psutil)"
+## (Within `git bash`) assert that Python scripts work.
+git map-branches
+```
+
+Run this sequence through the following upgrade/downgrade procedures:
 
-Python installs are sourced from gs://chrome-infra/python276_bin.zip .
+* Cold default installation.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Run through test steps.
+  - Test upgrade to bleeding edge (if it differs).
+    - Run `python.bat` in another shell, keep it open
+    - Add `.bleeding_edge` to `depot_tools` root.
+    - Run through test steps.
+    - In the old `python.bat` shell, run `import psutil`, confirm that it
+      works.
+    - Close the Python shell, run `gclient version`, ensure that old directory
+      is cleaned.
+* Cold bleeding edge installation.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Add `.bleeding_edge` to `depot_tools` root.
+  - Run through test steps.
+  - Test downgrade to default (if it differs).
+    - Run `python.bat` in another shell, keep it open
+    - Delete `.bleeding_edge` from `depot_tools` root.
+    - Run through test steps.
+    - In the old `python.bat` shell, run `import psutil`, confirm that it
+      works.
+    - Close the Python shell, run `gclient version`, ensure that old directory
+      is cleaned.
+* Warm bleeding edge upgrade.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Run `gclient version` to load defaults.
+  - Run `python.bat` in another shell, keep it open
+  - Add `.bleeding_edge` to `depot_tools` root.
+  - Run through test steps.
+  - In the old `python.bat` shell, run `import psutil`, confirm that it
+    works.
+  - Close the Python shell, run `gclient version`, ensure that old directory is
+    cleaned.
 
-The process to create them is sort-of-documented in the README of the python
-zip file.
+This will take some time, but will ensure that all affected bots and users
+should not encounter any problems due to the change. As systems and users are
+migrated off of this implicit bootstrap, the testing procedure will become less
+critical.

bootstrap/win/git-bash.template.sh

@@ -2,11 +2,11 @@
 export EDITOR=${EDITOR:=notepad}
 WIN_BASE=`dirname $0`
 UNIX_BASE=`cygpath "$WIN_BASE"`
-export PATH=$PATH:$UNIX_BASE/SVN_BIN_DIR:$UNIX_BASE/PYTHON_BIN_DIR:$UNIX_BASE/PYTHON_BIN_DIR/Scripts
+export PATH="$PATH:$UNIX_BASE/${PYTHON_BIN_RELDIR_UNIX}:$UNIX_BASE/${PYTHON_BIN_RELDIR_UNIX}/Scripts"
 export PYTHON_DIRECT=1
 export PYTHONUNBUFFERED=1
 if [[ $# > 0 ]]; then
-  $UNIX_BASE/GIT_BIN_DIR/bin/bash.exe "$@"
+  $UNIX_BASE/${GIT_BIN_RELDIR_UNIX}/bin/bash.exe "$@"
 else
-  $UNIX_BASE/GIT_BIN_DIR/git-bash.exe &
+  $UNIX_BASE/${GIT_BIN_RELDIR_UNIX}/git-bash.exe &
 fi

bootstrap/win/git.template.bat

 @echo off
 setlocal
 if not defined EDITOR set EDITOR=notepad
-set PATH=%~dp0GIT_BIN_DIR\cmd;%~dp0;%PATH%
-"%~dp0GIT_BIN_DIR\GIT_PROGRAM" %*
+set PATH=%~dp0${GIT_BIN_RELDIR}\cmd;%~dp0;%PATH%
+"%~dp0${GIT_BIN_RELDIR}\${GIT_PROGRAM}" %*

bootstrap/win/manifest_bleeding_edge.txt

+# CIPD manifest for Windows tools.
+#
+# We must install anything that we want included on PATH to a different
+# subdirectory than Git, as Git's msys bash strips its root directory
+# from PATH, hence the subdirs.
+#
+# If any paths or package layouts change, updates will be required in
+# "win_tools.bat" and "win_tools.py" templates.
+#
+# "win_tools.bat" has a hard requirement that the Python package contains the
+# string "cpython" and ends with the CIPD tag "version:VERSION". It uses this
+# to extract VERSION.
+
+@Subdir python
+infra/python/cpython/windows-386 version:2.7.6
+
+@Subdir git
+infra/git/${platform} version:2.10.0

bootstrap/win/python27.new.bat

+@echo off
+:: Copyright 2017 The Chromium Authors. All rights reserved.
+:: Use of this source code is governed by a BSD-style license that can be
+:: found in the LICENSE file.
+
+setlocal
+set PYTHON_BAT_RUNNER=1
+
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:: This file is automatically generated by "bootstrap\win\win_tools.py", and
+:: should not be modified.
+::
+:: The previous "::" block acts as a nop-sled. Each time a batch file executes
+:: a command, it reloads itself and seeks to its previous execution offset to
+:: begin execution. Updating this batch file is, therefore, risky, since any
+:: running Python instance that is using the old batch file will reload the new
+:: batch file once the Python command terminates and resume at some unknown
+:: offset.
+::
+:: With the sled in place, a previous instance will resume mid-label. We are
+:: assuming that the offset of the Python invocation is greater than the
+:: PYTHON_BAT_RUNNER set command, which is the case since the old instance has
+:: a large PATH set block before the Python execution. Old instances will hit
+:: the next block of code without PYTHON_BAT_RUNNER set. New instances will have
+:: it set.
+::
+:: We remedy this in the future by having the batch file load its core paths
+:: from an external file via "set /p", removing the need to modify "python.bat"
+:: during upgrade.
+::
+:: After all of the old batch files are believed to be replaced, we can remove
+:: the PYTHON_BAT_RUNNER block and the sled. For this update, old instances
+:: will resume past the end of the file and terminate.
+
+if not "%PYTHON_BAT_RUNNER%" == "1" goto :END
+
+set /p PYTHON_BIN_RELDIR=<%~dp0python_bin_reldir.txt
+set PATH=%~dp0%PYTHON_BIN_RELDIR%;%~dp0%PYTHON_BIN_RELDIR%\Scripts;%PATH%
+"%~dp0%PYTHON_BIN_RELDIR%\python.exe" %*
+
+:END

bootstrap/win/win_tools.bat

@@ -4,10 +4,11 @@
 :: found in the LICENSE file.
 
 :: This script will determine if python or git binaries need updates. It
-:: returns 123 if the user's shell must restart, otherwise !0 is failure
+:: returns !0 as failure
 
-:: Sadly, we can't use SETLOCAL here otherwise it ERRORLEVEL is not correctly
-:: returned.
+:: Note: we set EnableDelayedExpansion so we can perform string manipulations
+:: in our manifest parsing loop. This only works on Windows XP+.
+setlocal EnableDelayedExpansion
 
 set CHROME_INFRA_URL=https://storage.googleapis.com/chrome-infra/
 :: It used to be %~dp0 but ADODB.Stream may fail to write to this directory if
@@ -19,31 +20,90 @@ pushd %~dp0..\..
 set WIN_TOOLS_ROOT_DIR=%CD%
 popd
 
+:: Extra arguments to pass to our "win_tools.py" script.
+set WIN_TOOLS_EXTRA_ARGS=
+set WIN_TOOLS_PYTHON_BIN=%WIN_TOOLS_ROOT_DIR%\python.bat
+
+:: TODO: Deprecate this when legacy mode is disabled.
 if "%1" == "force" (
-  set WIN_TOOLS_FORCE=1
+  set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --force
   shift /1
 )
 
+:: Determine if we're running a bleeding-edge installation.
+if not exist "%WIN_TOOLS_ROOT_DIR%\.bleeding_edge" (
+  set CIPD_MANIFEST=
+) else (
+  set CIPD_MANIFEST=manifest_bleeding_edge.txt
+  set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --bleeding-edge
+)
+
+:: Identify our CIPD executable. If the client executable exists, use it
+:: directly; otherwise, use "cipd.bat" to bootstrap the client. This
+:: optimization is useful because this script can be run frequently, and
+:: reduces execution time noticeably.
+::
+:: See "//cipd.bat" and "//cipd.ps1" for more information.
+set CIPD_EXE=%WIN_TOOLS_ROOT_DIR%\.cipd_client.exe
+if not exist "%CIPD_EXE%" set CIPD_EXE=%WIN_TOOLS_ROOT_DIR%\cipd.bat
+set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --cipd-client "%CIPD_EXE%"
+
+:: TODO: This logic will change when we deprecate legacy mode. For now, we
+:: assume !bleeding_edge == legacy.
+if "%CIPD_MANIFEST%" == "" goto :PY27_LEGACY_CHECK
 
-:PYTHON_CHECK
-:: Support revert from https://chromium-review.googlesource.com/c/563036
+:: We are committed to CIPD, and will use "win_tools.py" to perform our
+:: finalization.
 ::
-:: If the "python.bat" from that CL is installed, we will not know to
-:: replace it if the CL is reverted. To support this, we will actively
-:: destroy our "python.bat" if we detect a "python_bin_reldir.txt" file
-:: present, causing us to reinstall Python.
-if exist "%WIN_TOOLS_ROOT_DIR%\python_bin_reldir.txt" (
-  call copy /y "%~dp0python276.new.bat" "%WIN_TOOLS_ROOT_DIR%\python.bat" 1>nul
-  del "%WIN_TOOLS_ROOT_DIR%\python_bin_reldir.txt"
+:: Parse our CIPD manifest and identify the "cpython" version. We do this by
+:: reading it line-by-line, identifying the line containing "cpython", and
+:: stripping all text preceding "version:". This leaves us with the version
+:: string.
+::
+:: This method requires EnableDelayedExpansion, and extracts the Python version
+:: from our CIPD manifest. Variables referenced using "!" instead of "%" are
+:: delayed expansion variables.
+for /F "tokens=*" %%A in (%~dp0%CIPD_MANIFEST%) do (
+  set LINE=%%A
+  if not "x!LINE:cpython=!" == "x!LINE!" set PYTHON_VERSION=!LINE:*version:=!
+)
+if "%PYTHON_VERSION%" == "" (
+  @echo Could not extract Python version from manifest.
+  set ERRORLEVEL=1
+  goto :END
 )
 
-if not exist "%WIN_TOOLS_ROOT_DIR%\python276_bin" goto :PY27_INSTALL
-if not exist "%WIN_TOOLS_ROOT_DIR%\python.bat" goto :PY27_INSTALL
-set ERRORLEVEL=0
-goto :GIT_CHECK
+:: We will take the version string, replace "." with "_", and surround it with
+:: "win-tools-<PYTHON_VERSION>_bin" so that it matches "win_tools.py"'s cleanup
+:: expression and ".gitignore".
+::
+:: We incorporate PYTHON_VERSION into the "win_tools" directory name so that
+:: new installations don't interfere with long-running Python processes if
+:: Python is upgraded.
+set WIN_TOOLS_NAME=win_tools-%PYTHON_VERSION:.=_%_bin
+set WIN_TOOLS_PATH=%WIN_TOOLS_ROOT_DIR%\%WIN_TOOLS_NAME%
+set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --win-tools-name "%WIN_TOOLS_NAME%"
+
+:: Install our CIPD packages.
+call "%CIPD_EXE%" ensure -ensure-file "%~dp0%CIPD_MANIFEST%" -root "%WIN_TOOLS_PATH%"
+if errorlevel 1 goto :END
 
+set WIN_TOOLS_PYTHON_BIN=%WIN_TOOLS_PATH%\python\bin\python.exe
+goto :WIN_TOOLS_PY
 
-:PY27_INSTALL
+
+:: LEGACY Support
+::
+:: This is a full Python installer. It falls through to "win_tools.py",
+:: instructing it to not handle Python installation. This should be removed
+:: once we commit to CIPD.
+:PY27_LEGACY_CHECK
+if not exist "%WIN_TOOLS_ROOT_DIR%\python.bat" goto :PY27_LEGACY_INSTALL
+if not exist "%WIN_TOOLS_ROOT_DIR%\python276_bin" goto :PY27_LEGACY_INSTALL
+goto :WIN_TOOLS_PY
+
+
+:PY27_LEGACY_INSTALL
 echo Installing python 2.7.6...
 :: Cleanup python directory if it was existing.
 set PYTHON_URL=%CHROME_INFRA_URL%python276_bin.zip
@@ -51,7 +111,7 @@ if exist "%WIN_TOOLS_ROOT_DIR%\python276_bin\." rd /q /s "%WIN_TOOLS_ROOT_DIR%\p
 if exist "%ZIP_DIR%\python276.zip" del "%ZIP_DIR%\python276.zip"
 echo Fetching from %PYTHON_URL%
 cscript //nologo //e:jscript "%~dp0get_file.js" %PYTHON_URL% "%ZIP_DIR%\python276_bin.zip"
-if errorlevel 1 goto :PYTHON_FAIL
+if errorlevel 1 goto :PYTHON_LEGACY_FAIL
 :: Will create python276_bin\...
 cscript //nologo //e:jscript "%~dp0unzip.js" "%ZIP_DIR%\python276_bin.zip" "%WIN_TOOLS_ROOT_DIR%"
 :: Create the batch files.
@@ -59,22 +119,25 @@ call copy /y "%~dp0python276.new.bat" "%WIN_TOOLS_ROOT_DIR%\python.bat" 1>nul
 call copy /y "%~dp0pylint.new.bat" "%WIN_TOOLS_ROOT_DIR%\pylint.bat" 1>nul
 del "%ZIP_DIR%\python276_bin.zip"
 set ERRORLEVEL=0
-goto :GIT_CHECK
+goto :WIN_TOOLS_PY
 
 
-:PYTHON_FAIL
+:PYTHON_LEGACY_FAIL
 echo ... Failed to checkout python automatically.
 echo You should get the "prebaked" version at %PYTHON_URL%
 set ERRORLEVEL=1
 goto :END
 
-:GIT_CHECK
-"%WIN_TOOLS_ROOT_DIR%\python.bat" "%~dp0git_bootstrap.py"
-goto :END
 
-:returncode
-set WIN_TOOLS_ROOT_DIR=
-exit /b %ERRORLEVEL%
+:: This executes "win_tools.py" using the WIN_TOOLS_PYTHON_BIN Python
+:: interpreter.
+:WIN_TOOLS_PY
+call "%WIN_TOOLS_PYTHON_BIN%" "%~dp0win_tools.py" %WIN_TOOLS_EXTRA_ARGS%
+
 
 :END
-call :returncode %ERRORLEVEL%
+set EXPORT_ERRORLEVEL=%ERRORLEVEL%
+endlocal & (
+  set ERRORLEVEL=%EXPORT_ERRORLEVEL%
+)
+exit /b %ERRORLEVEL%

bootstrap/win/git_bootstrap.py → bootstrap/win/win_tools.py

@@ -3,12 +3,16 @@
 # found in the LICENSE file.
 
 import argparse
+import collections
 import contextlib
 import fnmatch
+import hashlib
 import logging
 import os
 import platform
+import posixpath
 import shutil
+import string
 import subprocess
 import sys
 import tempfile
@@ -21,7 +25,6 @@ DEVNULL = open(os.devnull, 'w')
 
 BAT_EXT = '.bat' if sys.platform.startswith('win') else ''
 
-
 # Top-level stubs to generate that fall through to executables within the Git
 # directory.
 STUBS = {
@@ -32,13 +35,134 @@ STUBS = {
 }
 
 
-def _check_call(argv, input=None, **kwargs):
+# Accumulated template parameters for generated stubs.
+class Template(collections.namedtuple('Template', (
+    'PYTHON_RELDIR', 'PYTHON_BIN_RELDIR', 'PYTHON_BIN_RELDIR_UNIX',
+    'GIT_BIN_RELDIR', 'GIT_BIN_RELDIR_UNIX', 'GIT_PROGRAM',
+    ))):
+
+  @classmethod
+  def empty(cls):
+    return cls(**{k: None for k in cls._fields})
+
+  def maybe_install(self, name, dst_path):
+    """Installs template |name| to |dst_path| if it has changed.
+
+    This loads the template |name| from THIS_DIR, resolves template parameters,
+    and installs it to |dst_path|. See `maybe_update` for more information.
+
+    Args:
+      name (str): The name of the template to install.
+      dst_path (str): The destination filesystem path.
+
+    Returns (bool): True if |dst_path| was updated, False otherwise.
+    """
+    template_path = os.path.join(THIS_DIR, name)
+    with open(template_path, 'r') as fd:
+      t = string.Template(fd.read())
+    return maybe_update(t.safe_substitute(self._asdict()), dst_path)
+
+
+def maybe_update(content, dst_path):
+  """Writes |content| to |dst_path| if |dst_path| does not already match.
+
+  This function will ensure that there is a file at |dst_path| containing
+  |content|. If |dst_path| already exists and contains |content|, no operation
+  will be performed, preserving filesystem modification times and avoiding
+  potential write contention.
+
+  Args:
+    content (str): The file content.
+    dst_path (str): The destination filesystem path.
+
+  Returns (bool): True if |dst_path| was updated, False otherwise.
+  """
+  # If the path already exists and matches the new content, refrain from writing
+  # a new one.
+  if os.path.exists(dst_path):
+    with open(dst_path, 'r') as fd:
+      if fd.read() == content:
+        return False
+
+  logging.debug('Updating %r', dst_path)
+  with open(dst_path, 'w') as fd:
+    fd.write(content)
+  return True
+
+
+def maybe_copy(src_path, dst_path):
+  """Writes the content of |src_path| to |dst_path| if needed.
+
+  See `maybe_update` for more information.
+
+  Args:
+    src_path (str): The content source filesystem path.
+    dst_path (str): The destination filesystem path.
+
+  Returns (bool): True if |dst_path| was updated, False otherwise.
+  """
+  with open(src_path, 'r') as fd:
+    content = fd.read()
+  return maybe_update(content, dst_path)
+
+
+def call_if_outdated(stamp_path, stamp_version, fn):
+  """Invokes |fn| if the stamp at |stamp_path| doesn't match |stamp_version|.
+
+  This can be used to keep a filesystem record of whether an operation has been
+  performed. The record is stored at |stamp_path|. To invalidate a record,
+  change the value of |stamp_version|.
+
+  After |fn| completes successfully, |stamp_path| will be updated to match
+  |stamp_version|, preventing the same update from happening in the future.
+
+  Args:
+    stamp_path (str): The filesystem path of the stamp file.
+    stamp_version (str): The desired stamp version.
+    fn (callable): A callable to invoke if the current stamp version doesn't
+        match |stamp_version|.
+
+  Returns (bool): True if an update occurred.
+  """
+
+  stamp_version = stamp_version.strip()
+  if os.path.isfile(stamp_path):
+    with open(stamp_path, 'r') as fd:
+      current_version = fd.read().strip()
+    if current_version == stamp_version:
+      return False
+
+  fn()
+
+  with open(stamp_path, 'w') as fd:
+    fd.write(stamp_version)
+  return True
+
+
+def _in_use(path):
+  """Checks if a Windows file is in use.
+
+  When Windows is using an executable, it prevents other writers from
+  modifying or deleting that executable. We can safely test for an in-use
+  file by opening it in write mode and checking whether or not there was
+  an error.
+
+  Returns (bool): True if the file was in use, False if not.
+  """
+  try:
+    with open(path, 'r+'):
+      return False
+  except IOError:
+    return True
+
+
+def _check_call(argv, stdin_input=None, **kwargs):
   """Wrapper for subprocess.check_call that adds logging."""
   logging.info('running %r', argv)
-  if input is not None:
+  if stdin_input is not None:
     kwargs['stdin'] = subprocess.PIPE
   proc = subprocess.Popen(argv, **kwargs)
-  proc.communicate(input=input)
+  proc.communicate(input=stdin_input)
   if proc.returncode:
     raise subprocess.CalledProcessError(proc.returncode, argv, None)
 
@@ -83,8 +207,7 @@ def get_os_bitness():
 
 def get_target_git_version(args):
   """Returns git version that should be installed."""
-  if (args.bleeding_edge or
-      os.path.exists(os.path.join(ROOT_DIR, '.git_bleeding_edge'))):
+  if args.bleeding_edge:
     git_version_file = 'git_version_bleeding_edge.txt'
   else:
     git_version_file = 'git_version.txt'
@@ -101,10 +224,38 @@ def clean_up_old_git_installations(git_directory, force):
       _safe_rmtree(full_entry)
 
 
-def cipd_install(args, dest_directory, package, version):
-  """Installs CIPD |package| at |version| into |dest_directory|."""
+def clean_up_old_installations(skip_dir):
+  """Removes Python installations other than |skip_dir|.
+
+  This includes an "in-use" check against the "python.exe" in a given directory
+  to avoid removing Python executables that are currently ruinning. We need
+  this because our Python bootstrap may be run after (and by) other software
+  that is using the bootstrapped Python!
+  """
+  root_contents = os.listdir(ROOT_DIR)
+  for f in ('win_tools-*_bin', 'python27*_bin', 'git-*_bin'):
+    for entry in fnmatch.filter(root_contents, f):
+      full_entry = os.path.join(ROOT_DIR, entry)
+      if full_entry == skip_dir or not os.path.isdir(full_entry):
+        continue
+
+      logging.info('Cleaning up old installation %r', entry)
+      for python_exe in (
+          os.path.join(full_entry, 'python', 'bin', 'python.exe'), # CIPD
+          os.path.join(full_entry, 'python.exe'), # Legacy ZIP distributions.
+          ):
+        if os.path.isfile(python_exe) and _in_use(python_exe):
+          logging.info('Python executable %r is in-use; skipping', python_exe)
+          break
+      else:
+        _safe_rmtree(full_entry)
+
+
+def cipd_ensure(args, dest_directory, package, version):
+  """Installs a CIPD package using "ensure"."""
   logging.info('Installing CIPD package %r @ %r', package, version)
-  manifest = '%s %s\n' % (package, version)
+  manifest_text = '%s %s\n' % (package, version)
+
   cipd_args = [
     args.cipd_client,
     'ensure',
@@ -115,7 +266,7 @@ def cipd_install(args, dest_directory, package, version):
     cipd_args.extend(['-cache-dir', args.cipd_cache_directory])
   if args.verbose:
     cipd_args.append('-verbose')
-  _check_call(cipd_args, input=manifest)
+  _check_call(cipd_args, stdin_input=manifest_text)
 
 
 def need_to_install_git(args, git_directory, legacy):
@@ -159,10 +310,9 @@ def need_to_install_git(args, git_directory, legacy):
 def install_git_legacy(args, git_version, git_directory, cipd_platform):
   _safe_rmtree(git_directory)
   with _tempdir() as temp_dir:
-    cipd_install(args,
-                 temp_dir,
-                 'infra/depot_tools/git_installer/%s' % cipd_platform,
-                 'v' + git_version.replace('.', '_'))
+    cipd_ensure(args, temp_dir,
+        package='infra/depot_tools/git_installer/%s' % cipd_platform,
+        version='v' + git_version.replace('.', '_'))
 
     # 7-zip has weird expectations for command-line syntax. Pass it as a string
     # to avoid subprocess module quoting breaking it. Also double-escape
@@ -188,26 +338,9 @@ def install_git(args, git_version, git_directory, legacy):
       logging.info('Deleting legacy Git directory: %s', git_directory)
       _safe_rmtree(git_directory)
 
-    cipd_install(
-        args,
-        git_directory,
-        'infra/git/%s' % (cipd_platform,),
-        git_version)
-
-  # Create Git templates and configure its base layout.
-  with open(os.path.join(THIS_DIR, 'git.template.bat')) as f:
-    git_template = f.read()
-  git_template = git_template.replace(
-      'GIT_BIN_DIR', os.path.relpath(git_directory, ROOT_DIR))
-
-  for stub_name, relpath in STUBS.iteritems():
-    with open(os.path.join(ROOT_DIR, stub_name), 'w') as f:
-      f.write(git_template.replace('GIT_PROGRAM', relpath))
-  with open(os.path.join(THIS_DIR, 'git-bash.template.sh')) as f:
-    git_bash_template = f.read()
-  with open(os.path.join(ROOT_DIR, 'git-bash'), 'w') as f:
-    f.write(git_bash_template.replace(
-        'GIT_BIN_DIR', os.path.relpath(git_directory, ROOT_DIR)))
+    cipd_ensure(args, git_directory,
+        package='infra/git/%s' % (cipd_platform,),
+        version=git_version)
 
   if legacy:
     # The non-legacy Git bundle includes "python.sh".
@@ -217,55 +350,158 @@ def install_git(args, git_version, git_directory, legacy):
         os.path.join(THIS_DIR, 'profile.d.python.sh'),
         os.path.join(git_directory, 'etc', 'profile.d', 'python.sh'))
 
-  git_bat_path = os.path.join(ROOT_DIR, 'git.bat')
-  _check_call([git_bat_path, 'config', '--system', 'core.autocrlf', 'false'])
-  _check_call([git_bat_path, 'config', '--system', 'core.filemode', 'false'])
-  _check_call([git_bat_path, 'config', '--system', 'core.preloadindex', 'true'])
-  _check_call([git_bat_path, 'config', '--system', 'core.fscache', 'true'])
-
-
-def main(argv):
-  parser = argparse.ArgumentParser()
-  parser.add_argument('--bits', type=int, choices=(32,64),
-                      help='Bitness of the client to install. Default on this'
-                      ' system: %(default)s', default=get_os_bitness())
-  parser.add_argument('--cipd-client',
-                      help='Path to CIPD client binary. default: %(default)s',
-                      default=os.path.join(ROOT_DIR, 'cipd'+BAT_EXT))
-  parser.add_argument('--cipd-cache-directory',
-                      help='Path to CIPD cache directory.')
-  parser.add_argument('--bleeding-edge', action='store_true',
-                      help='Force bleeding edge Git.')
-  parser.add_argument('--force', action='store_true',
-                      help='Always re-install git.')
-  parser.add_argument('--verbose', action='store_true')
-  args = parser.parse_args(argv)
-
-  if os.environ.get('WIN_TOOLS_FORCE') == '1':
-    args.force = True
-
-  logging.basicConfig(level=logging.INFO if args.verbose else logging.WARN)
 
+def ensure_git(args, template):
   git_version = get_target_git_version(args)
 
   git_directory_tag = git_version.split(':')
   git_directory = os.path.join(
       ROOT_DIR, 'git-%s-%s_bin' % (git_directory_tag[-1], args.bits))
-  git_docs_dir = os.path.join(
-      git_directory, 'mingw%s' % args.bits, 'share', 'doc', 'git-doc')
 
   clean_up_old_git_installations(git_directory, args.force)
 
+  git_bin_dir = os.path.relpath(git_directory, ROOT_DIR)
+  template = template._replace(
+      GIT_BIN_RELDIR=git_bin_dir,
+      GIT_BIN_RELDIR_UNIX=git_bin_dir)
+
   # Modern Git versions use CIPD tags beginning with "version:". If the tag
   # does not begin with that, use the legacy installer.
   legacy = not git_version.startswith('version:')
   if need_to_install_git(args, git_directory, legacy):
     install_git(args, git_version, git_directory, legacy)
 
+  git_postprocess(template, git_directory)
+
+  return template
+
+
+# Version of "git_postprocess" system configuration (see |git_postprocess|).
+GIT_POSTPROCESS_VERSION = '1'
+
+
+def git_get_mingw_dir(git_directory):
+  """Returns (str) The "mingw" directory in a Git installation, or None."""
+  for candidate in ('mingw64', 'mingw32'):
+    mingw_dir = os.path.join(git_directory, candidate)
+    if os.path.isdir(mingw_dir):
+      return mingw_dir
+  return None
+
+
+def git_postprocess(template, git_directory):
   # Update depot_tools files for "git help <command>"
-  docsrc = os.path.join(ROOT_DIR, 'man', 'html')
-  for name in os.listdir(docsrc):
-    shutil.copy2(os.path.join(docsrc, name), os.path.join(git_docs_dir, name))
+  mingw_dir = git_get_mingw_dir(git_directory)
+  if mingw_dir:
+    docsrc = os.path.join(ROOT_DIR, 'man', 'html')
+    git_docs_dir = os.path.join(mingw_dir, 'share', 'doc', 'git-doc')
+    for name in os.listdir(docsrc):
+      maybe_copy(
+          os.path.join(docsrc, name),
+          os.path.join(git_docs_dir, name))
+  else:
+    logging.info('Could not find mingw directory for %r.', git_directory)
+
+  # Create Git templates and configure its base layout.
+  for stub_name, relpath in STUBS.iteritems():
+    stub_template = template._replace(GIT_PROGRAM=relpath)
+    stub_template.maybe_install(
+        'git.template.bat',
+        os.path.join(ROOT_DIR, stub_name))
+
+  # Set-up our system configuration environment. The following set of
+  # parameters is versioned by "GIT_POSTPROCESS_VERSION". If they change,
+  # update "GIT_POSTPROCESS_VERSION" accordingly.
+  def configure_git_system():
+    git_bat_path = os.path.join(ROOT_DIR, 'git.bat')
+    _check_call([git_bat_path, 'config', '--system', 'core.autocrlf', 'false'])
+    _check_call([git_bat_path, 'config', '--system', 'core.filemode', 'false'])
+    _check_call([git_bat_path, 'config', '--system', 'core.preloadindex',
+                 'true'])
+    _check_call([git_bat_path, 'config', '--system', 'core.fscache', 'true'])
+
+  call_if_outdated(
+      os.path.join(git_directory, '.git_postprocess'),
+      GIT_POSTPROCESS_VERSION,
+      configure_git_system)
+
+
+def main(argv):
+  parser = argparse.ArgumentParser()
+  parser.add_argument('--verbose', action='store_true')
+  parser.add_argument('--win-tools-name',
+                      help='The directory of the Python installation. '
+                           '(legacy) If missing, use legacy Windows tools '
+                           'processing')
+  parser.add_argument('--bleeding-edge', action='store_true',
+                      help='Force bleeding edge Git.')
+
+  group = parser.add_argument_group('legacy flags')
+  group.add_argument('--force', action='store_true',
+                     help='Always re-install everything.')
+  group.add_argument('--bits', type=int, choices=(32,64),
+                     help='Bitness of the client to install. Default on this'
+                     ' system: %(default)s', default=get_os_bitness())
+  group.add_argument('--cipd-client',
+                     help='Path to CIPD client binary. default: %(default)s',
+                     default=os.path.join(ROOT_DIR, 'cipd'+BAT_EXT))
+  group.add_argument('--cipd-cache-directory',
+                     help='Path to CIPD cache directory.')
+  args = parser.parse_args(argv)
+
+  logging.basicConfig(level=logging.DEBUG if args.verbose else logging.WARN)
+
+  template = Template.empty()
+  if not args.win_tools_name:
+    # Legacy support.
+    template = template._replace(
+        PYTHON_RELDIR='python276_bin',
+        PYTHON_BIN_RELDIR='python276_bin',
+        PYTHON_BIN_RELDIR_UNIX='python276_bin')
+    template = ensure_git(args, template)
+  else:
+    template = template._replace(
+        PYTHON_RELDIR=os.path.join(args.win_tools_name, 'python'),
+        PYTHON_BIN_RELDIR=os.path.join(args.win_tools_name, 'python', 'bin'),
+        PYTHON_BIN_RELDIR_UNIX=posixpath.join(
+            args.win_tools_name, 'python', 'bin'),
+        GIT_BIN_RELDIR=os.path.join(args.win_tools_name, 'git'),
+        GIT_BIN_RELDIR_UNIX=posixpath.join(args.win_tools_name, 'git'))
+
+    win_tools_dir = os.path.join(ROOT_DIR, args.win_tools_name)
+    git_postprocess(template, os.path.join(win_tools_dir, 'git'))
+
+    # Clean up any old Python installations.
+    clean_up_old_installations(win_tools_dir)
+
+  # Emit our Python bin depot-tools-relative directory. This is ready by
+  # "python.bat" to identify the path of the current Python installation.
+  #
+  # We use this indirection so that upgrades can change this pointer to
+  # redirect "python.bat" to a new Python installation. We can't just update
+  # "python.bat" because batch file executions reload the batch file and seek
+  # to the previous cursor in between every command, so changing the batch
+  # file contents could invalidate any existing executions.
+  #
+  # The intention is that the batch file itself never needs to change when
+  # switching Python versions.
+  maybe_update(
+      template.PYTHON_BIN_RELDIR,
+      os.path.join(ROOT_DIR, 'python_bin_reldir.txt'))
+
+  # Install our "python.bat" shim.
+  # TODO: Move this to generic shim installation once legacy support is
+  # removed and this code path is the only one.
+  template.maybe_install(
+      'python27.new.bat',
+      os.path.join(ROOT_DIR, 'python.bat'))
+
+  # Re-evaluate and regenerate our root templated files.
+  for src_name, dst_name in (
+      ('git-bash.template.sh', 'git-bash'),
+      ('pylint.new.bat', 'pylint.bat'),
+      ):
+    template.maybe_install(src_name, os.path.join(ROOT_DIR, dst_name))
 
   return 0