Skip to content

C
Chromium_Depot_tools
Commit d6d5c46c authored by Gavin Mak's avatar Gavin Mak Committed by LUCI CQ
Browse files
Options

Add PYTHON_VERSION_COMPATIBILITY to gitiles and bot_update 

Setting PYTHON_VERSION_COMPATIBILITY = PY2+3 makes "recipe.py test" run
tests in both py2 and py3.

Bug: 1227140
Change-Id: I56a47f0491cbade172e1dc1239cbf6c2a0faa64f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3116850
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: 's avatarJosip Sokcevic <sokcevic@google.com>
parent 5d0fbd6a
Show whitespace changes
Inline Side-by-side
Showing with 67 additions and 42 deletions
recipes/README.recipes.md
View file @ d6d5c46c
......@@ -3,14 +3,14 @@
## Table of Contents
**[Recipe Modules](#Recipe-Modules)**
* [bot_update](#recipe_modules-bot_update) &mdash; Recipe module to ensure a checkout is consistent on a bot.
* [bot_update](#recipe_modules-bot_update) (Python3 ✅) &mdash; Recipe module to ensure a checkout is consistent on a bot.
* [cipd](#recipe_modules-cipd) &mdash; API for interacting with CIPD.
* [depot_tools](#recipe_modules-depot_tools) &mdash; The `depot_tools` module provides safe functions to access paths within the depot_tools repo.
* [gclient](#recipe_modules-gclient)
* [gerrit](#recipe_modules-gerrit)
* [git](#recipe_modules-git)
* [git_cl](#recipe_modules-git_cl)
* [gitiles](#recipe_modules-gitiles)
* [gitiles](#recipe_modules-gitiles) (Python3 ✅)
* [gsutil](#recipe_modules-gsutil)
* [osx_sdk](#recipe_modules-osx_sdk) &mdash; The `osx_sdk` module provides safe functions to access a semi-hermetic XCode installation.
* [presubmit](#recipe_modules-presubmit)
......@@ -18,9 +18,9 @@
* [windows_sdk](#recipe_modules-windows_sdk) &mdash; The `windows_sdk` module provides safe functions to access a hermetic Microsoft Visual Studio installation.
**[Recipes](#Recipes)**
* [bot_update:examples/full](#recipes-bot_update_examples_full)
* [bot_update:tests/do_not_retry_patch_failures_in_cq](#recipes-bot_update_tests_do_not_retry_patch_failures_in_cq)
* [bot_update:tests/ensure_checkout](#recipes-bot_update_tests_ensure_checkout)
* [bot_update:examples/full](#recipes-bot_update_examples_full) (Python3 ✅)
* [bot_update:tests/do_not_retry_patch_failures_in_cq](#recipes-bot_update_tests_do_not_retry_patch_failures_in_cq) (Python3 ✅)
* [bot_update:tests/ensure_checkout](#recipes-bot_update_tests_ensure_checkout) (Python3 ✅)
* [cipd:examples/full](#recipes-cipd_examples_full)
* [cipd:examples/platform_suffix](#recipes-cipd_examples_platform_suffix)
* [depot_tools:examples/full](#recipes-depot_tools_examples_full)
......@@ -32,8 +32,8 @@
* [gerrit:examples/full](#recipes-gerrit_examples_full)
* [git:examples/full](#recipes-git_examples_full)
* [git_cl:examples/full](#recipes-git_cl_examples_full)
* [gitiles:examples/full](#recipes-gitiles_examples_full)
* [gitiles:tests/parse_repo_url](#recipes-gitiles_tests_parse_repo_url)
* [gitiles:examples/full](#recipes-gitiles_examples_full) (Python3 ✅)
* [gitiles:tests/parse_repo_url](#recipes-gitiles_tests_parse_repo_url) (Python3 ✅)
* [gsutil:examples/full](#recipes-gsutil_examples_full)
* [osx_sdk:examples/full](#recipes-osx_sdk_examples_full)
* [presubmit:examples/full](#recipes-presubmit_examples_full)
......@@ -48,9 +48,9 @@
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/commit\_position][recipe_engine/recipe_modules/commit_position], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/milo][recipe_engine/recipe_modules/milo], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#3): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/commit\_position][recipe_engine/recipe_modules/commit_position], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/milo][recipe_engine/recipe_modules/milo], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
Recipe module to ensure a checkout is consistent on a bot.
......@@ -622,20 +622,20 @@ DEPRECATED. Consider using gerrit.get_change_description instead.
&mdash; **def [upload](/recipes/recipe_modules/git_cl/api.py#42)(self, message, upload_args=None, \*\*kwargs):**
### *recipe_modules* / [gitiles](/recipes/recipe_modules/gitiles)
[DEPS](/recipes/recipe_modules/gitiles/__init__.py#5): [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/url][recipe_engine/recipe_modules/url]
[DEPS](/recipes/recipe_modules/gitiles/__init__.py#7): [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/url][recipe_engine/recipe_modules/url]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
#### **class [Gitiles](/recipes/recipe_modules/gitiles/api.py#17)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
#### **class [Gitiles](/recipes/recipe_modules/gitiles/api.py#18)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
Module for polling a git repository using the Gitiles web interface.
&mdash; **def [canonicalize\_repo\_url](/recipes/recipe_modules/gitiles/api.py#222)(self, repo_url):**
&mdash; **def [canonicalize\_repo\_url](/recipes/recipe_modules/gitiles/api.py#227)(self, repo_url):**
Returns a canonical form of repo_url. If not recognized, returns as is.
&mdash; **def [commit\_log](/recipes/recipe_modules/gitiles/api.py#120)(self, url, commit, step_name=None, attempts=None):**
&mdash; **def [commit\_log](/recipes/recipe_modules/gitiles/api.py#121)(self, url, commit, step_name=None, attempts=None):**
Returns: (dict) the Gitiles commit log structure for a given commit.
......@@ -645,7 +645,7 @@ Args:
* step_name (str): If not None, override the step name.
* attempts (int): Number of times to try the request before failing.
&mdash; **def [download\_archive](/recipes/recipe_modules/gitiles/api.py#162)(self, repository_url, destination, revision='refs/heads/main'):**
&mdash; **def [download\_archive](/recipes/recipe_modules/gitiles/api.py#167)(self, repository_url, destination, revision='refs/heads/main'):**
Downloads an archive of the repo and extracts it to `destination`.
......@@ -661,7 +661,7 @@ Args:
* revision (str): The ref or revision in the repo to download. Defaults to
'refs/heads/main'.
&mdash; **def [download\_file](/recipes/recipe_modules/gitiles/api.py#136)(self, repository_url, file_path, branch='main', step_name=None, attempts=None, \*\*kwargs):**
&mdash; **def [download\_file](/recipes/recipe_modules/gitiles/api.py#137)(self, repository_url, file_path, branch='main', step_name=None, attempts=None, \*\*kwargs):**
Downloads raw file content from a Gitiles repository.
......@@ -675,7 +675,7 @@ Args:
Returns:
Raw file content.
&mdash; **def [log](/recipes/recipe_modules/gitiles/api.py#74)(self, url, ref, limit=0, cursor=None, step_name=None, attempts=None, \*\*kwargs):**
&mdash; **def [log](/recipes/recipe_modules/gitiles/api.py#75)(self, url, ref, limit=0, cursor=None, step_name=None, attempts=None, \*\*kwargs):**
Returns the most recent commits under the given ref with properties.
......@@ -698,17 +698,17 @@ Returns:
Cursor can be used for subsequent calls to log for paging. If None,
signals that there are no more commits to fetch.
&mdash; **def [parse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#211)(self, repo_url):**
&mdash; **def [parse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#216)(self, repo_url):**
Returns (host, project) pair.
Returns (None, None) if repo_url is not recognized.
&mdash; **def [refs](/recipes/recipe_modules/gitiles/api.py#62)(self, url, step_name='refs', attempts=None):**
&mdash; **def [refs](/recipes/recipe_modules/gitiles/api.py#63)(self, url, step_name='refs', attempts=None):**
Returns a list of refs in the remote repository.
&mdash; **def [unparse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#218)(self, host, project):**
&mdash; **def [unparse\_repo\_url](/recipes/recipe_modules/gitiles/api.py#223)(self, host, project):**
Generates a Gitiles repo URL. See also parse_repo_url.
### *recipe_modules* / [gsutil](/recipes/recipe_modules/gsutil)
......@@ -1057,25 +1057,25 @@ Raises:
### *recipes* / [bot\_update:examples/full](/recipes/recipe_modules/bot_update/examples/full.py)
[DEPS](/recipes/recipe_modules/bot_update/examples/full.py#5): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
[DEPS](/recipes/recipe_modules/bot_update/examples/full.py#7): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/examples/full.py#23)(api):**
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/examples/full.py#25)(api):**
### *recipes* / [bot\_update:tests/do\_not\_retry\_patch\_failures\_in\_cq](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py)
[DEPS](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#5): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#17)(api):**
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py#19)(api):**
### *recipes* / [bot\_update:tests/ensure\_checkout](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py)
[DEPS](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#7): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/json][recipe_engine/recipe_modules/json]
[DEPS](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#9): [bot\_update](#recipe_modules-bot_update), [gclient](#recipe_modules-gclient), [recipe\_engine/json][recipe_engine/recipe_modules/json]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#14)(api):**
&mdash; **def [RunSteps](/recipes/recipe_modules/bot_update/tests/ensure_checkout.py#16)(api):**
### *recipes* / [cipd:examples/full](/recipes/recipe_modules/cipd/examples/full.py)
[DEPS](/recipes/recipe_modules/cipd/examples/full.py#8): [cipd](#recipe_modules-cipd), [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
......@@ -1155,18 +1155,18 @@ PYTHON_VERSION_COMPATIBILITY: PY2
&mdash; **def [RunSteps](/recipes/recipe_modules/git_cl/examples/full.py#17)(api):**
### *recipes* / [gitiles:examples/full](/recipes/recipe_modules/gitiles/examples/full.py)
[DEPS](/recipes/recipe_modules/gitiles/examples/full.py#5): [gitiles](#recipe_modules-gitiles), [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/gitiles/examples/full.py#7): [gitiles](#recipe_modules-gitiles), [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/examples/full.py#14)(api):**
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/examples/full.py#16)(api):**
### *recipes* / [gitiles:tests/parse\_repo\_url](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py)
[DEPS](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#5): [gitiles](#recipe_modules-gitiles), [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#7): [gitiles](#recipe_modules-gitiles), [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2
PYTHON_VERSION_COMPATIBILITY: PY2+3
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#12)(api):**
&mdash; **def [RunSteps](/recipes/recipe_modules/gitiles/tests/parse_repo_url.py#14)(api):**
### *recipes* / [gsutil:examples/full](/recipes/recipe_modules/gsutil/examples/full.py)
[DEPS](/recipes/recipe_modules/gsutil/examples/full.py#5): [gsutil](#recipe_modules-gsutil), [recipe\_engine/path][recipe_engine/recipe_modules/path]
......
recipes/recipe_modules/bot_update/__init__.py
View file @ d6d5c46c
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'depot_tools',
'gclient',
......
recipes/recipe_modules/bot_update/examples/full.py
View file @ d6d5c46c
......@@ -2,6 +2,8 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'bot_update',
'gclient',
......
recipes/recipe_modules/bot_update/test_api.py
View file @ d6d5c46c
......@@ -100,11 +100,11 @@ class BotUpdateTestApi(recipe_test_api.RecipeTestApi):
@staticmethod
def gen_revision(project):
"""Hash project to bogus deterministic git hash values."""
h = hashlib.sha1(project)
h = hashlib.sha1(project.encode('utf-8'))
return h.hexdigest()
@staticmethod
def gen_commit_position(project):
"""Hash project to bogus deterministic Cr-Commit-Position values."""
h = hashlib.sha1(project)
h = hashlib.sha1(project.encode('utf-8'))
return struct.unpack('!I', h.digest()[:4])[0] % 300000
recipes/recipe_modules/bot_update/tests/do_not_retry_patch_failures_in_cq.py
View file @ d6d5c46c
......@@ -2,6 +2,10 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine import post_process
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'bot_update',
'gclient',
......@@ -11,8 +15,6 @@ DEPS = [
'recipe_engine/step',
]
from recipe_engine import post_process
def RunSteps(api):
src_cfg = api.gclient.make_config()
......
recipes/recipe_modules/bot_update/tests/ensure_checkout.py
View file @ d6d5c46c
......@@ -4,6 +4,8 @@
from recipe_engine import post_process
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'bot_update',
'gclient',
......
recipes/recipe_modules/gitiles/__init__.py
View file @ d6d5c46c
......@@ -2,6 +2,8 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'recipe_engine/json',
'recipe_engine/path',
......
recipes/recipe_modules/gitiles/api.py
View file @ d6d5c46c
......@@ -5,6 +5,7 @@
from __future__ import division
import base64
import sys
try:
import urlparse
......@@ -157,7 +158,11 @@ class Gitiles(recipe_api.RecipeApi):
**kwargs)
if step_result.json.output['value'] is None:
return None
return base64.b64decode(step_result.json.output['value'])
# TODO(crbug.com/1227140): Clean up when py2 is no longer supported.
value = base64.b64decode(step_result.json.output['value'])
if sys.version_info >= (3,):
return value.decode('utf-8')
return value
def download_archive(self, repository_url, destination,
revision='refs/heads/main'):
......
recipes/recipe_modules/gitiles/examples/full.py
View file @ d6d5c46c
......@@ -2,6 +2,8 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'gitiles',
'recipe_engine/json',
......
recipes/recipe_modules/gitiles/test_api.py
View file @ d6d5c46c
......@@ -87,9 +87,15 @@ class GitilesTestApi(recipe_test_api.RecipeTestApi):
return d
def make_hash(self, *bases):
return hashlib.sha1(':'.join(bases)).hexdigest()
return hashlib.sha1(':'.join(bases).encode('utf-8')).hexdigest()
def make_encoded_file(self, data):
value = None
# TODO(crbug.com/1227140): Clean up when py2 is no longer supported.
try:
value = base64.b64encode(data.encode('utf-8')).decode('utf-8')
except UnicodeDecodeError: #pragma: nocover
value = base64.b64encode(data)
return self.m.json.output({
'value': base64.b64encode(data),
'value': value,
})
recipes/recipe_modules/gitiles/tests/parse_repo_url.py
View file @ d6d5c46c
......@@ -2,6 +2,8 @@
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = 'PY2+3'
DEPS = [
'gitiles',
'recipe_engine/properties',
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment