[win_tools] Use bundled Python CIPD packages.

Enable bundled Python CIPD packages in bleeding-edge mode. This
replaces the ZIP unpacking approach used before, and introduces
validation and management through the CIPD tool. The bleeding edge
version will only install if a sentinel file is present in the
"depot_tools" root; otherwise, default behavior will continue.

This method adds a upgrade and downgrade path to/from ZIP and
CIPD installations. This is done by rewriting the "win_tools.bat"
process:

1) Ensure that a bootstrap Python is present.
2) Use it to run "win_tools.py", which has the functionality of
    "git_bootstrap.py" plus Python installation.
3) Run "win_tools.py" with appropriate flags.

Some tricks were employed to handle cases where there is an
already-running Python instance that uses the current Python
installation and executable. This happens on bots because the
system uses the same "depot_tools" checkout at multiple launch
layers. To this end, we use the "python.bat" as the "current Python"
authority and refrain from cleaning up old Python directories if their
"python.exe" binaries are currently in use.

We change the Git bleeding edge file to share the same
sentinel file as Python, ".bleeding_edge".

The new Python should have the same facilities as the original Python
bundle.

BUG=chromium:740171
TEST=local

Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736
Reviewed-on: https://chromium-review.googlesource.com/563036
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

.gitignore

@@ -14,6 +14,7 @@
 /git.bat
 /gitk.bat
 /pylint.bat
+/.bleeding_edge
 /.pylint.d/
 /python
 /python.bat
@@ -22,7 +23,7 @@
 /svn
 /svn.bat
 /svnversion.bat
-/.bleeding_edge
+/python_bin_reldir.txt
 /.codereview_upload_cookies
 /.gitconfig
 /.git_bleeding_edge
@@ -33,6 +34,7 @@
 
 # Ignore locations where third-party tools are placed during bootstrapping.
 /python*_bin
+/win_tools*_bin
 /git_bin
 /git-*_bin
 /svn_bin

bootstrap/win/README.md

@@ -3,6 +3,14 @@
 This directory has the 'magic' for the `depot_tools` windows binary update
 mechanisms.
 
+A previous Python may actually be in use when it is run, preventing us
+from replacing it outright without breaking running code. To
+ommodate this, and Python cleanup, we handle Python in two stages:
+
+1. Use CIPD to install both Git and Python at once.
+2. Use "win_tools.py" as a post-processor to install generated files and
+   fix-ups.
+
 ## Software bootstrapped
   * Python (https://www.python.org/)
   * Git for Windows (https://git-for-windows.github.io/)
@@ -18,31 +26,28 @@ work.
 package is present, and if so, if it's the expected version. If either of those
 cases is not true, it will download and unpack the respective binary.
 
-Downloading is done with [get_file.js](./get_file.js), which is a windows script
-host javascript utility to vaguely impersonate `wget`.
+Installation of Git and Python is done by the [win_tools.bat](./win_tools.bat)
+script, which uses CIPD (via the [cipd](/cipd.bat) bootstrap) to acquire and
+install each package into the root of the `depot_tools` repository. Afterwards,
+the [win_tools.py](./win_tools.py) Python script is invoked to install stubs,
+wrappers, and support scripts into `depot_tools` for end-users.
 
-Through a comedy of history, each binary is stored and retrieved differently.
+### Manifest
 
-### Git
+The Git and Python versions are specified in [manifest.txt](./manifest.txt).
 
-Git installs are mirrored versions of the official Git-for-Windows Portable
-releases.
-  * Original: `https://github.com/git-for-windows/git/releases`
-  * Mirror: `gs://chrome-infra/PortableGit*.7z.exe`
+There is an associated file,
+[manifest_bleeding_edge.txt](./manifest_bleeding_edge.txt), that can be used
+to canary new versions on select bots. Any bots with a `.bleeding_edge` file
+in their `depot_tools` root will automatically use the bleeding edge manifest.
+This allows opt-in systems to test against new versions of Python or Git. Once
+those versions have been verified correct, `manifest.txt` can be updated to the
+same specification, which will cause the remainder of systems to update.
 
-#### Updating git version
-  1. Download the new `PortableGit-X.Y.Z-{32,64}.7z.exe` from the
-     git-for-windows release page.
-  1. From either console.developers.google.com or the CLI, do:
-    1. Upload those to the gs://chrome-infra Google Storage bucket.
-    1. Set the `allUsers Reader` permission (click the "Public link" checkbox
-       next to the binaries).
-  1. Edit the `git_version.txt` or `git_version_bleeding_edge.txt` file to
-     be the new version.
-    1. You can use the bleeding edge version to get early feedback/stage a
-       rollout/etc. Users can select this version by 'touch'ing the
-       `.git_bleeding_edge` file in the root depot_tools directory.
-  1. Commit the CL.
+### Bundles
+
+Git and Python bundle construction is documented in
+[infra packaging](https://chromium.googlesource.com/infra/infra/+/master/doc/packaging/).
 
 Note that in order for the update to take effect, `gclient` currently needs to
 run twice. The first time it will update the `depot_tools` repo, and the second
@@ -50,9 +55,79 @@ time it will see the new git version and update to it. This is a bug that should
 be fixed, in case you're reading this and this paragraph infuriates you more
 than the rest of this README.
 
-### Python
+## Testing
+
+After any modification to this script set, a test sequence should be run on a
+Windows bot.
+
+The post-processing will regenerate "python.bat" to point to the current
+Python instance. Any previous Python installations will stick around, but
+new invocations will use the new instance. Old installations will die
+off either due to processes terminating or systems restarting. When this
+happens, they will be cleaned up by the post-processing script.
+
+Testing
+=======
+
+For each of the following test scenarios, run these commands and verify that
+they are working:
+
+```bash
+# Assert that `gclient` invocation will update (and do the update).
+gclient version
+
+# Assert that Python fundamentally works.
+python -c "import psutil; dir(psutil)"
+
+# Assert that Python scripts work from `cmd.exe`.
+git map-branches
+
+# Assert that `git bash` works.
+git bash
+
+## (Within `git bash`) assert that Python fundamentally works.
+python -c "import psutil; dir(psutil)"
+## (Within `git bash`) assert that Python scripts work.
+git map-branches
+```
+
+Run this sequence through the following upgrade/downgrade procedures:
 
-Python installs are sourced from gs://chrome-infra/python276_bin.zip .
+* Cold default installation.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Run through test steps.
+  - Test upgrade to bleeding edge (if it differs).
+    - Run `python.bat` in another shell, keep it open
+    - Add `.bleeding_edge` to `depot_tools` root.
+    - Run through test steps.
+    - In the old `python.bat` shell, run `import psutil`, confirm that it
+      works.
+    - Close the Python shell, run `gclient version`, ensure that old directory
+      is cleaned.
+* Cold bleeding edge installation.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Add `.bleeding_edge` to `depot_tools` root.
+  - Run through test steps.
+  - Test downgrade to default (if it differs).
+    - Run `python.bat` in another shell, keep it open
+    - Delete `.bleeding_edge` from `depot_tools` root.
+    - Run through test steps.
+    - In the old `python.bat` shell, run `import psutil`, confirm that it
+      works.
+    - Close the Python shell, run `gclient version`, ensure that old directory
+      is cleaned.
+* Warm bleeding edge upgrade.
+  - Clean `depot_tools` via: `git clean -x -f -d .`
+  - Run `gclient version` to load defaults.
+  - Run `python.bat` in another shell, keep it open
+  - Add `.bleeding_edge` to `depot_tools` root.
+  - Run through test steps.
+  - In the old `python.bat` shell, run `import psutil`, confirm that it
+    works.
+  - Close the Python shell, run `gclient version`, ensure that old directory is
+    cleaned.
 
-The process to create them is sort-of-documented in the README of the python
-zip file.
+This will take some time, but will ensure that all affected bots and users
+should not encounter any problems due to the change. As systems and users are
+migrated off of this implicit bootstrap, the testing procedure will become less
+critical.

bootstrap/win/git-bash.template.sh

@@ -2,11 +2,11 @@
 export EDITOR=${EDITOR:=notepad}
 WIN_BASE=`dirname $0`
 UNIX_BASE=`cygpath "$WIN_BASE"`
-export PATH=$PATH:$UNIX_BASE/SVN_BIN_DIR:$UNIX_BASE/PYTHON_BIN_DIR:$UNIX_BASE/PYTHON_BIN_DIR/Scripts
+export PATH="$PATH:$UNIX_BASE/${PYTHON_BIN_RELDIR_UNIX}:$UNIX_BASE/${PYTHON_BIN_RELDIR_UNIX}/Scripts"
 export PYTHON_DIRECT=1
 export PYTHONUNBUFFERED=1
 if [[ $# > 0 ]]; then
-  $UNIX_BASE/GIT_BIN_DIR/bin/bash.exe "$@"
+  $UNIX_BASE/${GIT_BIN_RELDIR_UNIX}/bin/bash.exe "$@"
 else
-  $UNIX_BASE/GIT_BIN_DIR/git-bash.exe &
+  $UNIX_BASE/${GIT_BIN_RELDIR_UNIX}/git-bash.exe &
 fi

bootstrap/win/git.template.bat

 @echo off
 setlocal
 if not defined EDITOR set EDITOR=notepad
-set PATH=%~dp0GIT_BIN_DIR\cmd;%~dp0;%PATH%
-"%~dp0GIT_BIN_DIR\GIT_PROGRAM" %*
+set PATH=%~dp0${GIT_BIN_RELDIR}\cmd;%~dp0;%PATH%
+"%~dp0${GIT_BIN_RELDIR}\${GIT_PROGRAM}" %*

bootstrap/win/manifest_bleeding_edge.txt

+# CIPD manifest for Windows tools.
+#
+# We must install anything that we want included on PATH to a different
+# subdirectory than Git, as Git's msys bash strips its root directory
+# from PATH, hence the subdirs.
+#
+# If any paths or package layouts change, updates will be required in
+# "win_tools.bat" and "win_tools.py" templates.
+#
+# "win_tools.bat" has a hard requirement that the Python package contains the
+# string "cpython" and ends with the CIPD tag "version:VERSION". It uses this
+# to extract VERSION.
+
+@Subdir python
+infra/python/cpython/windows-386 version:2.7.6
+
+@Subdir git
+infra/git/${platform} version:2.10.0

bootstrap/win/python27.new.bat

+@echo off
+:: Copyright 2017 The Chromium Authors. All rights reserved.
+:: Use of this source code is governed by a BSD-style license that can be
+:: found in the LICENSE file.
+
+setlocal
+set PYTHON_BAT_RUNNER=1
+
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:: This file is automatically generated by "bootstrap\win\win_tools.py", and
+:: should not be modified.
+::
+:: The previous "::" block acts as a nop-sled. Each time a batch file executes
+:: a command, it reloads itself and seeks to its previous execution offset to
+:: begin execution. Updating this batch file is, therefore, risky, since any
+:: running Python instance that is using the old batch file will reload the new
+:: batch file once the Python command terminates and resume at some unknown
+:: offset.
+::
+:: With the sled in place, a previous instance will resume mid-label. We are
+:: assuming that the offset of the Python invocation is greater than the
+:: PYTHON_BAT_RUNNER set command, which is the case since the old instance has
+:: a large PATH set block before the Python execution. Old instances will hit
+:: the next block of code without PYTHON_BAT_RUNNER set. New instances will have
+:: it set.
+::
+:: We remedy this in the future by having the batch file load its core paths
+:: from an external file via "set /p", removing the need to modify "python.bat"
+:: during upgrade.
+::
+:: After all of the old batch files are believed to be replaced, we can remove
+:: the PYTHON_BAT_RUNNER block and the sled. For this update, old instances
+:: will resume past the end of the file and terminate.
+
+if not "%PYTHON_BAT_RUNNER%" == "1" goto :END
+
+set /p PYTHON_BIN_RELDIR=<%~dp0python_bin_reldir.txt
+set PATH=%~dp0%PYTHON_BIN_RELDIR%;%~dp0%PYTHON_BIN_RELDIR%\Scripts;%PATH%
+"%~dp0%PYTHON_BIN_RELDIR%\python.exe" %*
+
+:END

bootstrap/win/win_tools.bat

@@ -4,10 +4,11 @@
 :: found in the LICENSE file.
 
 :: This script will determine if python or git binaries need updates. It
-:: returns 123 if the user's shell must restart, otherwise !0 is failure
+:: returns !0 as failure
 
-:: Sadly, we can't use SETLOCAL here otherwise it ERRORLEVEL is not correctly
-:: returned.
+:: Note: we set EnableDelayedExpansion so we can perform string manipulations
+:: in our manifest parsing loop. This only works on Windows XP+.
+setlocal EnableDelayedExpansion
 
 set CHROME_INFRA_URL=https://storage.googleapis.com/chrome-infra/
 :: It used to be %~dp0 but ADODB.Stream may fail to write to this directory if
@@ -19,31 +20,90 @@ pushd %~dp0..\..
 set WIN_TOOLS_ROOT_DIR=%CD%
 popd
 
+:: Extra arguments to pass to our "win_tools.py" script.
+set WIN_TOOLS_EXTRA_ARGS=
+set WIN_TOOLS_PYTHON_BIN=%WIN_TOOLS_ROOT_DIR%\python.bat
+
+:: TODO: Deprecate this when legacy mode is disabled.
 if "%1" == "force" (
-  set WIN_TOOLS_FORCE=1
+  set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --force
   shift /1
 )
 
+:: Determine if we're running a bleeding-edge installation.
+if not exist "%WIN_TOOLS_ROOT_DIR%\.bleeding_edge" (
+  set CIPD_MANIFEST=
+) else (
+  set CIPD_MANIFEST=manifest_bleeding_edge.txt
+  set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --bleeding-edge
+)
+
+:: Identify our CIPD executable. If the client executable exists, use it
+:: directly; otherwise, use "cipd.bat" to bootstrap the client. This
+:: optimization is useful because this script can be run frequently, and
+:: reduces execution time noticeably.
+::
+:: See "//cipd.bat" and "//cipd.ps1" for more information.
+set CIPD_EXE=%WIN_TOOLS_ROOT_DIR%\.cipd_client.exe
+if not exist "%CIPD_EXE%" set CIPD_EXE=%WIN_TOOLS_ROOT_DIR%\cipd.bat
+set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --cipd-client "%CIPD_EXE%"
+
+:: TODO: This logic will change when we deprecate legacy mode. For now, we
+:: assume !bleeding_edge == legacy.
+if "%CIPD_MANIFEST%" == "" goto :PY27_LEGACY_CHECK
 
-:PYTHON_CHECK
-:: Support revert from https://chromium-review.googlesource.com/c/563036
+:: We are committed to CIPD, and will use "win_tools.py" to perform our
+:: finalization.
 ::
-:: If the "python.bat" from that CL is installed, we will not know to
-:: replace it if the CL is reverted. To support this, we will actively
-:: destroy our "python.bat" if we detect a "python_bin_reldir.txt" file
-:: present, causing us to reinstall Python.
-if exist "%WIN_TOOLS_ROOT_DIR%\python_bin_reldir.txt" (
-  call copy /y "%~dp0python276.new.bat" "%WIN_TOOLS_ROOT_DIR%\python.bat" 1>nul
-  del "%WIN_TOOLS_ROOT_DIR%\python_bin_reldir.txt"
+:: Parse our CIPD manifest and identify the "cpython" version. We do this by
+:: reading it line-by-line, identifying the line containing "cpython", and
+:: stripping all text preceding "version:". This leaves us with the version
+:: string.
+::
+:: This method requires EnableDelayedExpansion, and extracts the Python version
+:: from our CIPD manifest. Variables referenced using "!" instead of "%" are
+:: delayed expansion variables.
+for /F "tokens=*" %%A in (%~dp0%CIPD_MANIFEST%) do (
+  set LINE=%%A
+  if not "x!LINE:cpython=!" == "x!LINE!" set PYTHON_VERSION=!LINE:*version:=!
+)
+if "%PYTHON_VERSION%" == "" (
+  @echo Could not extract Python version from manifest.
+  set ERRORLEVEL=1
+  goto :END
 )
 
-if not exist "%WIN_TOOLS_ROOT_DIR%\python276_bin" goto :PY27_INSTALL
-if not exist "%WIN_TOOLS_ROOT_DIR%\python.bat" goto :PY27_INSTALL
-set ERRORLEVEL=0
-goto :GIT_CHECK
+:: We will take the version string, replace "." with "_", and surround it with
+:: "win-tools-<PYTHON_VERSION>_bin" so that it matches "win_tools.py"'s cleanup
+:: expression and ".gitignore".
+::
+:: We incorporate PYTHON_VERSION into the "win_tools" directory name so that
+:: new installations don't interfere with long-running Python processes if
+:: Python is upgraded.
+set WIN_TOOLS_NAME=win_tools-%PYTHON_VERSION:.=_%_bin
+set WIN_TOOLS_PATH=%WIN_TOOLS_ROOT_DIR%\%WIN_TOOLS_NAME%
+set WIN_TOOLS_EXTRA_ARGS=%WIN_TOOLS_EXTRA_ARGS% --win-tools-name "%WIN_TOOLS_NAME%"
+
+:: Install our CIPD packages.
+call "%CIPD_EXE%" ensure -ensure-file "%~dp0%CIPD_MANIFEST%" -root "%WIN_TOOLS_PATH%"
+if errorlevel 1 goto :END
 
+set WIN_TOOLS_PYTHON_BIN=%WIN_TOOLS_PATH%\python\bin\python.exe
+goto :WIN_TOOLS_PY
 
-:PY27_INSTALL
+
+:: LEGACY Support
+::
+:: This is a full Python installer. It falls through to "win_tools.py",
+:: instructing it to not handle Python installation. This should be removed
+:: once we commit to CIPD.
+:PY27_LEGACY_CHECK
+if not exist "%WIN_TOOLS_ROOT_DIR%\python.bat" goto :PY27_LEGACY_INSTALL
+if not exist "%WIN_TOOLS_ROOT_DIR%\python276_bin" goto :PY27_LEGACY_INSTALL
+goto :WIN_TOOLS_PY
+
+
+:PY27_LEGACY_INSTALL
 echo Installing python 2.7.6...
 :: Cleanup python directory if it was existing.
 set PYTHON_URL=%CHROME_INFRA_URL%python276_bin.zip
@@ -51,7 +111,7 @@ if exist "%WIN_TOOLS_ROOT_DIR%\python276_bin\." rd /q /s "%WIN_TOOLS_ROOT_DIR%\p
 if exist "%ZIP_DIR%\python276.zip" del "%ZIP_DIR%\python276.zip"
 echo Fetching from %PYTHON_URL%
 cscript //nologo //e:jscript "%~dp0get_file.js" %PYTHON_URL% "%ZIP_DIR%\python276_bin.zip"
-if errorlevel 1 goto :PYTHON_FAIL
+if errorlevel 1 goto :PYTHON_LEGACY_FAIL
 :: Will create python276_bin\...
 cscript //nologo //e:jscript "%~dp0unzip.js" "%ZIP_DIR%\python276_bin.zip" "%WIN_TOOLS_ROOT_DIR%"
 :: Create the batch files.
@@ -59,22 +119,25 @@ call copy /y "%~dp0python276.new.bat" "%WIN_TOOLS_ROOT_DIR%\python.bat" 1>nul
 call copy /y "%~dp0pylint.new.bat" "%WIN_TOOLS_ROOT_DIR%\pylint.bat" 1>nul
 del "%ZIP_DIR%\python276_bin.zip"
 set ERRORLEVEL=0
-goto :GIT_CHECK
+goto :WIN_TOOLS_PY
 
 
-:PYTHON_FAIL
+:PYTHON_LEGACY_FAIL
 echo ... Failed to checkout python automatically.
 echo You should get the "prebaked" version at %PYTHON_URL%
 set ERRORLEVEL=1
 goto :END
 
-:GIT_CHECK
-"%WIN_TOOLS_ROOT_DIR%\python.bat" "%~dp0git_bootstrap.py"
-goto :END
 
-:returncode
-set WIN_TOOLS_ROOT_DIR=
-exit /b %ERRORLEVEL%
+:: This executes "win_tools.py" using the WIN_TOOLS_PYTHON_BIN Python
+:: interpreter.
+:WIN_TOOLS_PY
+call "%WIN_TOOLS_PYTHON_BIN%" "%~dp0win_tools.py" %WIN_TOOLS_EXTRA_ARGS%
+
 
 :END
-call :returncode %ERRORLEVEL%
+set EXPORT_ERRORLEVEL=%ERRORLEVEL%
+endlocal & (
+  set ERRORLEVEL=%EXPORT_ERRORLEVEL%
+)
+exit /b %ERRORLEVEL%