[recipe_modules] Copy over windows_sdk from infra.git

Change-Id: Ibc528126b26c4f7b7f2e308c89390878bfcfde86
Reviewed-on: https://chromium-review.googlesource.com/1166207
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

recipes/README.recipes.md

@@ -15,6 +15,7 @@
   * [infra_paths](#recipe_modules-infra_paths)
   * [presubmit](#recipe_modules-presubmit)
   * [tryserver](#recipe_modules-tryserver)
+  * [windows_sdk](#recipe_modules-windows_sdk) — The `windows_sdk` module provides safe functions to access a hermetic Microsoft Visual Studio installation.
 
 **[Recipes](#Recipes)**
   * [bot_update:examples/buildbucket](#recipes-bot_update_examples_buildbucket)
@@ -33,6 +34,7 @@
   * [infra_paths:examples/full](#recipes-infra_paths_examples_full)
   * [presubmit:examples/full](#recipes-presubmit_examples_full)
   * [tryserver:examples/full](#recipes-tryserver_examples_full)
+  * [windows_sdk:examples/full](#recipes-windows_sdk_examples_full)
 ## Recipe Modules
 
 ### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
@@ -743,6 +745,32 @@ Mark the tryjob result as a test failure.
 
 This means we started running actual tests (not prerequisite steps
 like checkout or compile), and some of these tests have failed.
+### *recipe_modules* / [windows\_sdk](/recipes/recipe_modules/windows_sdk)
+
+[DEPS](/recipes/recipe_modules/windows_sdk/__init__.py#5): [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+
+The `windows_sdk` module provides safe functions to access a hermetic
+Microsoft Visual Studio installation.
+
+Available only to Google-run bots.
+
+#### **class [WindowsSDKApi](/recipes/recipe_modules/windows_sdk/api.py#15)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
+
+API for using Windows SDK distributed via CIPD.
+
+&emsp; **@contextmanager**<br>&mdash; **def [\_\_call\_\_](/recipes/recipe_modules/windows_sdk/api.py#23)(self, path=None, version=None, enabled=True):**
+
+Setups the SDK environment when enabled.
+
+Args:
+  path (path): Path to a directory where to install the SDK
+    (default is '[start_dir]/windows_sdk')
+  version (str): CIPD version of the SDK
+    (default is set via $infra/windows_sdk.version property)
+  enabled (bool): Whether the SDK should be used or not.
+
+Raises:
+    StepFailure or InfraFailure.
 ## Recipes
 
 ### *recipes* / [bot\_update:examples/buildbucket](/recipes/recipe_modules/bot_update/examples/buildbucket.py)
@@ -827,8 +855,14 @@ Move things around in a loop!
 [DEPS](/recipes/recipe_modules/tryserver/examples/full.py#5): [tryserver](#recipe_modules-tryserver), [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
 
 &mdash; **def [RunSteps](/recipes/recipe_modules/tryserver/examples/full.py#17)(api):**
+### *recipes* / [windows\_sdk:examples/full](/recipes/recipe_modules/windows_sdk/examples/full.py)
+
+[DEPS](/recipes/recipe_modules/windows_sdk/examples/full.py#5): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+
+&mdash; **def [RunSteps](/recipes/recipe_modules/windows_sdk/examples/full.py#13)(api):**
 
 [recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1f25d6536d9972411f4540f2e883e1560ddeb0ad/README.recipes.md#recipe_modules-buildbucket
+[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1f25d6536d9972411f4540f2e883e1560ddeb0ad/README.recipes.md#recipe_modules-cipd
 [recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1f25d6536d9972411f4540f2e883e1560ddeb0ad/README.recipes.md#recipe_modules-context
 [recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1f25d6536d9972411f4540f2e883e1560ddeb0ad/README.recipes.md#recipe_modules-file
 [recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1f25d6536d9972411f4540f2e883e1560ddeb0ad/README.recipes.md#recipe_modules-json

recipes/recipe_modules/windows_sdk/__init__.py

+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+DEPS = [
+    'recipe_engine/cipd',
+    'recipe_engine/context',
+    'recipe_engine/json',
+    'recipe_engine/path',
+    'recipe_engine/platform',
+    'recipe_engine/step',
+]
+
+from recipe_engine.recipe_api import Property
+from recipe_engine.config import ConfigGroup, Single
+
+PROPERTIES = {
+  '$depot_tools/windows_sdk': Property(
+    help='Properties specifically for the infra windows_sdk module.',
+    param_name='sdk_properties',
+    kind=ConfigGroup(
+      # CIPD instance ID, tag or ref for the Windows SDK version.
+      version=Single(str),
+    ), default={'version': 'uploaded:2018-06-13'},
+  )
+}

recipes/recipe_modules/windows_sdk/api.py

+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""The `windows_sdk` module provides safe functions to access a hermetic
+Microsoft Visual Studio installation.
+
+Available only to Google-run bots."""
+
+from contextlib import contextmanager
+
+from recipe_engine import recipe_api
+
+
+class WindowsSDKApi(recipe_api.RecipeApi):
+  """API for using Windows SDK distributed via CIPD."""
+
+  def __init__(self, sdk_properties, *args, **kwargs):
+    super(WindowsSDKApi, self).__init__(*args, **kwargs)
+
+    self._sdk_properties = sdk_properties
+
+  @contextmanager
+  def __call__(self, path=None, version=None, enabled=True):
+    """Setups the SDK environment when enabled.
+
+    Args:
+      path (path): Path to a directory where to install the SDK
+        (default is '[start_dir]/windows_sdk')
+      version (str): CIPD version of the SDK
+        (default is set via $infra/windows_sdk.version property)
+      enabled (bool): Whether the SDK should be used or not.
+
+    Raises:
+        StepFailure or InfraFailure.
+    """
+    if enabled:
+      sdk_dir = self._ensure_sdk(
+          path or self.m.path['start_dir'].join('windows_sdk'),
+          version or self._sdk_properties['version'])
+      try:
+        with self.m.context(**self._sdk_env(sdk_dir)):
+          yield
+      finally:
+        # cl.exe automatically starts background mspdbsrv.exe daemon which
+        # needs to be manually stopped so Swarming can tidy up after itself.
+        self.m.step('taskkill mspdbsrv',
+                    ['taskkill.exe', '/f', '/t', '/im', 'mspdbsrv.exe'])
+    else:
+      yield
+
+  def _ensure_sdk(self, sdk_dir, sdk_version):
+    """Ensures the Windows SDK CIPD package is installed.
+
+    Returns the directory where the SDK package has been installed.
+
+    Args:
+      path (path): Path to a directory.
+      version (str): CIPD instance ID, tag or ref.
+    """
+    with self.m.context(infra_steps=True):
+      pkgs = self.m.cipd.EnsureFile()
+      pkgs.add_package('chrome_internal/third_party/sdk/windows', sdk_version)
+      self.m.cipd.ensure(sdk_dir, pkgs)
+      return sdk_dir
+
+  def _sdk_env(self, sdk_dir):
+    """Constructs the environment for the SDK.
+
+    Returns environment and environment prefixes.
+
+    Args:
+      sdk_dir (path): Path to a directory containing the SDK.
+    """
+    env = {}
+    env_prefixes = {}
+
+    # Load .../win_sdk/bin/SetEnv.${arch}.json to extract the required
+    # environment. It contains a dict that looks like this:
+    # {
+    #   "env": {
+    #     "VAR": [["..", "..", "x"], ["..", "..", "y"]],
+    #     ...
+    #   }
+    # }
+    # All these environment variables need to be added to the environment
+    # for the compiler and linker to work.
+    filename = 'SetEnv.%s.json' % {32: 'x86', 64: 'x64'}[self.m.platform.bits]
+    step_result = self.m.json.read(
+        'read %s' % filename, sdk_dir.join('win_sdk', 'bin', filename),
+        step_test_data=lambda: self.m.json.test_api.output({
+            'env': {
+                'PATH': [['..', '..', 'win_sdk', 'bin', 'x64']],
+                'VSINSTALLDIR': [['..', '..\\']],
+            },
+        }))
+    data = step_result.json.output.get('env')
+    for key in data:
+      # recipes' Path() does not like .., ., \, or /, so this is cumbersome.
+      # What we want to do is:
+      #   [sdk_bin_dir.join(*e) for e in env[k]]
+      # Instead do that badly, and rely (but verify) on the fact that the paths
+      # are all specified relative to the root, but specified relative to
+      # win_sdk/bin (i.e. everything starts with "../../".)
+      results = []
+      for value in data[key]:
+        assert value[0] == '..' and (value[1] == '..' or value[1] == '..\\')
+        results.append('%s' % sdk_dir.join(*value[2:]))
+
+      # PATH is special-cased because we don't want to overwrite other things
+      # like C:\Windows\System32. Others are replacements because prepending
+      # doesn't necessarily makes sense, like VSINSTALLDIR.
+      if key.lower() == 'path':
+        env_prefixes[key] = results
+      else:
+        env[key] = ';'.join(results)
+
+    return {'env': env, 'env_prefixes': env_prefixes}

recipes/recipe_modules/windows_sdk/examples/full.expected/linux.json

+[
+  {
+    "cmd": [
+      "gn",
+      "gen",
+      "out/Release"
+    ],
+    "name": "gn"
+  },
+  {
+    "cmd": [
+      "ninja",
+      "-C",
+      "out/Release"
+    ],
+    "name": "ninja"
+  },
+  {
+    "name": "$result",
+    "recipe_result": null,
+    "status_code": 0
+  }
+]
\ No newline at end of file

recipes/recipe_modules/windows_sdk/examples/full.expected/mac.json

+[
+  {
+    "cmd": [
+      "gn",
+      "gen",
+      "out/Release"
+    ],
+    "name": "gn"
+  },
+  {
+    "cmd": [
+      "ninja",
+      "-C",
+      "out/Release"
+    ],
+    "name": "ninja"
+  },
+  {
+    "name": "$result",
+    "recipe_result": null,
+    "status_code": 0
+  }
+]
\ No newline at end of file

recipes/recipe_modules/windows_sdk/examples/full.expected/win.json

+[
+  {
+    "cmd": [
+      "cipd.bat",
+      "ensure",
+      "-root",
+      "[START_DIR]\\windows_sdk",
+      "-ensure-file",
+      "chrome_internal/third_party/sdk/windows uploaded:2018-06-13",
+      "-json-output",
+      "/path/to/tmp/json"
+    ],
+    "infra_step": true,
+    "name": "ensure_installed",
+    "~followup_annotations": [
+      "@@@STEP_LOG_LINE@json.output@{@@@",
+      "@@@STEP_LOG_LINE@json.output@  \"result\": {@@@",
+      "@@@STEP_LOG_LINE@json.output@    \"\": [@@@",
+      "@@@STEP_LOG_LINE@json.output@      {@@@",
+      "@@@STEP_LOG_LINE@json.output@        \"instance_id\": \"resolved-instance_id-of-uploaded:2018-06\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"package\": \"chrome_internal/third_party/sdk/windows\"@@@",
+      "@@@STEP_LOG_LINE@json.output@      }@@@",
+      "@@@STEP_LOG_LINE@json.output@    ]@@@",
+      "@@@STEP_LOG_LINE@json.output@  }@@@",
+      "@@@STEP_LOG_LINE@json.output@}@@@",
+      "@@@STEP_LOG_END@json.output@@@"
+    ]
+  },
+  {
+    "cmd": [
+      "python",
+      "-u",
+      "\nimport shutil\nimport sys\nshutil.copy(sys.argv[1], sys.argv[2])\n",
+      "[START_DIR]\\windows_sdk\\win_sdk\\bin\\SetEnv.x64.json",
+      "/path/to/tmp/json"
+    ],
+    "name": "read SetEnv.x64.json",
+    "~followup_annotations": [
+      "@@@STEP_LOG_LINE@json.output@{@@@",
+      "@@@STEP_LOG_LINE@json.output@  \"env\": {@@@",
+      "@@@STEP_LOG_LINE@json.output@    \"PATH\": [@@@",
+      "@@@STEP_LOG_LINE@json.output@      [@@@",
+      "@@@STEP_LOG_LINE@json.output@        \"..\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"..\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"win_sdk\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"bin\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"x64\"@@@",
+      "@@@STEP_LOG_LINE@json.output@      ]@@@",
+      "@@@STEP_LOG_LINE@json.output@    ], @@@",
+      "@@@STEP_LOG_LINE@json.output@    \"VSINSTALLDIR\": [@@@",
+      "@@@STEP_LOG_LINE@json.output@      [@@@",
+      "@@@STEP_LOG_LINE@json.output@        \"..\", @@@",
+      "@@@STEP_LOG_LINE@json.output@        \"..\\\\\"@@@",
+      "@@@STEP_LOG_LINE@json.output@      ]@@@",
+      "@@@STEP_LOG_LINE@json.output@    ]@@@",
+      "@@@STEP_LOG_LINE@json.output@  }@@@",
+      "@@@STEP_LOG_LINE@json.output@}@@@",
+      "@@@STEP_LOG_END@json.output@@@"
+    ]
+  },
+  {
+    "cmd": [
+      "gn",
+      "gen",
+      "out/Release"
+    ],
+    "env": {
+      "VSINSTALLDIR": "[START_DIR]\\windows_sdk"
+    },
+    "env_prefixes": {
+      "PATH": [
+        "[START_DIR]\\windows_sdk\\win_sdk\\bin\\x64"
+      ]
+    },
+    "name": "gn"
+  },
+  {
+    "cmd": [
+      "ninja",
+      "-C",
+      "out/Release"
+    ],
+    "env": {
+      "VSINSTALLDIR": "[START_DIR]\\windows_sdk"
+    },
+    "env_prefixes": {
+      "PATH": [
+        "[START_DIR]\\windows_sdk\\win_sdk\\bin\\x64"
+      ]
+    },
+    "name": "ninja"
+  },
+  {
+    "cmd": [
+      "taskkill.exe",
+      "/f",
+      "/t",
+      "/im",
+      "mspdbsrv.exe"
+    ],
+    "name": "taskkill mspdbsrv"
+  },
+  {
+    "name": "$result",
+    "recipe_result": null,
+    "status_code": 0
+  }
+]
\ No newline at end of file

recipes/recipe_modules/windows_sdk/examples/full.py

+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+DEPS = [
+  'windows_sdk',
+  'recipe_engine/platform',
+  'recipe_engine/properties',
+  'recipe_engine/step',
+]
+
+
+def RunSteps(api):
+  with api.windows_sdk(enabled=api.platform.is_win):
+    api.step('gn', ['gn', 'gen', 'out/Release'])
+    api.step('ninja', ['ninja', '-C', 'out/Release'])
+
+
+def GenTests(api):
+  for platform in ('linux', 'mac', 'win'):
+    properties = {
+        'buildername': 'test_builder',
+    }
+    yield (api.test(platform) +
+           api.platform.name(platform) +
+           api.properties.generic(**properties))