Skip to content

C
Chromium_Depot_tools
Commit 037f9b80 authored by Ryan Tseng's avatar Ryan Tseng Committed by Commit Bot
Browse files
Options

bot_update recipe: Upload source manifest 

Bug: 772529
Change-Id: I3f833740ddfbc71be42b140c99dcd4bbfeed54fe
Reviewed-on: https://chromium-review.googlesource.com/706071
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: 's avatarRobbie Iannucci <iannucci@chromium.org>
parent e6e0641b
Hide whitespace changes
Inline Side-by-side
Showing with 363 additions and 22 deletions
recipes/README.recipes.md
View file @ 037f9b80
...@@ -38,7 +38,7 @@ ...@@ -38,7 +38,7 @@
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update) ### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step] [DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/source\_manifest][recipe_engine/recipe_modules/source_manifest], [recipe\_engine/step][recipe_engine/recipe_modules/step]
Recipe module to ensure a checkout is consistent on a bot. Recipe module to ensure a checkout is consistent on a bot.
...@@ -50,12 +50,12 @@ Wrapper for easy calling of bot_update. ...@@ -50,12 +50,12 @@ Wrapper for easy calling of bot_update.
&mdash; **def [apply\_gerrit\_ref](/recipes/recipe_modules/bot_update/api.py#49)(self, root, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, gerrit_repo=None, gerrit_ref=None, step_name='apply_gerrit', \*\*kwargs):** &mdash; **def [apply\_gerrit\_ref](/recipes/recipe_modules/bot_update/api.py#49)(self, root, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, gerrit_repo=None, gerrit_ref=None, step_name='apply_gerrit', \*\*kwargs):**
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#434)(self, bot_update_step):** &mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#445)(self, bot_update_step):**
Deapplies a patch, taking care of DEPS and solution revisions properly. Deapplies a patch, taking care of DEPS and solution revisions properly.
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, \*\*kwargs):** &mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, manifest_name=None, \*\*kwargs):**
Args: Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot use_site_config_creds: If the oauth2 credentials are in the buildbot
...@@ -71,8 +71,10 @@ Args: ...@@ -71,8 +71,10 @@ Args:
disable_syntax_validation: (legacy) Disables syntax validation for DEPS. disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions, Needed as migration paths for recipes dealing with older revisions,
such as bisect. such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#411)(self, project_name, gclient_config=None):** &mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#422)(self, project_name, gclient_config=None):**
Returns all property names used for storing the checked-out revision of Returns all property names used for storing the checked-out revision of
a given project. a given project.
...@@ -839,6 +841,7 @@ Move things around in a loop! ...@@ -839,6 +841,7 @@ Move things around in a loop!
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-properties [recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-properties
[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-python [recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-python
[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-raw_io [recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-raw_io
[recipe_engine/recipe_modules/source_manifest]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-source_manifest
[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-step [recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-step
[recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-url [recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-url
[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/recipe_engine/recipe_api.py#992 [recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/recipe_engine/recipe_api.py#992
recipes/recipe_modules/bot_update/__init__.py
View file @ 037f9b80
...@@ -9,6 +9,7 @@ DEPS = [ ...@@ -9,6 +9,7 @@ DEPS = [
'recipe_engine/properties', 'recipe_engine/properties',
'recipe_engine/python', 'recipe_engine/python',
'recipe_engine/raw_io', 'recipe_engine/raw_io',
'recipe_engine/source_manifest',
'recipe_engine/step', 'recipe_engine/step',
'rietveld', 'rietveld',
'tryserver', 'tryserver',
......
recipes/recipe_modules/bot_update/api.py
View file @ 037f9b80
...@@ -77,7 +77,8 @@ class BotUpdateApi(recipe_api.RecipeApi): ...@@ -77,7 +77,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
root_solution_revision=None, rietveld=None, issue=None, root_solution_revision=None, rietveld=None, issue=None,
patchset=None, gerrit_no_reset=False, patchset=None, gerrit_no_reset=False,
gerrit_no_rebase_patch_ref=False, gerrit_no_rebase_patch_ref=False,
disable_syntax_validation=False, **kwargs): disable_syntax_validation=False, manifest_name=None,
**kwargs):
""" """
Args: Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot use_site_config_creds: If the oauth2 credentials are in the buildbot
...@@ -93,6 +94,8 @@ class BotUpdateApi(recipe_api.RecipeApi): ...@@ -93,6 +94,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
disable_syntax_validation: (legacy) Disables syntax validation for DEPS. disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions, Needed as migration paths for recipes dealing with older revisions,
such as bisect. such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
""" """
refs = refs or [] refs = refs or []
# We can re-use the gclient spec from the gclient module, since all the # We can re-use the gclient spec from the gclient module, since all the
...@@ -261,7 +264,6 @@ class BotUpdateApi(recipe_api.RecipeApi): ...@@ -261,7 +264,6 @@ class BotUpdateApi(recipe_api.RecipeApi):
root, first_sln, reverse_rev_map, self._fail_patch, root, first_sln, reverse_rev_map, self._fail_patch,
fixed_revisions=fixed_revisions) fixed_revisions=fixed_revisions)
# Add suffixes to the step name, if specified.
name = 'bot_update' name = 'bot_update'
if not patch: if not patch:
name += ' (without patch)' name += ' (without patch)'
...@@ -296,6 +298,15 @@ class BotUpdateApi(recipe_api.RecipeApi): ...@@ -296,6 +298,15 @@ class BotUpdateApi(recipe_api.RecipeApi):
step_text = result['step_text'] step_text = result['step_text']
step_result.presentation.step_text = step_text step_result.presentation.step_text = step_text
# Export the step results as a Source Manifest to LogDog.
if manifest_name:
if not patch:
# The param "patched" is purely cosmetic to mean "if false, this
# bot_update run exists purely to unpatch an existing patch".
manifest_name += '_unpatched'
self.m.source_manifest.set_json_manifest(
manifest_name, result.get('manifest', {}))
# Set the "checkout" path for the main solution. # Set the "checkout" path for the main solution.
# This is used by the Chromium module to figure out where to look for # This is used by the Chromium module to figure out where to look for
# the checkout. # the checkout.
......
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name.json 0 → 100644
View file @ 037f9b80
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]
\ No newline at end of file
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name_no_patch.json 0 → 100644
View file @ 037f9b80
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]
\ No newline at end of file
recipes/recipe_modules/bot_update/examples/full.py
View file @ 037f9b80
...@@ -49,6 +49,7 @@ def RunSteps(api): ...@@ -49,6 +49,7 @@ def RunSteps(api):
gerrit_no_reset = True if api.properties.get('gerrit_no_reset') else False gerrit_no_reset = True if api.properties.get('gerrit_no_reset') else False
gerrit_no_rebase_patch_ref = bool( gerrit_no_rebase_patch_ref = bool(
api.properties.get('gerrit_no_rebase_patch_ref')) api.properties.get('gerrit_no_rebase_patch_ref'))
manifest_name = api.properties.get('manifest_name')
if api.properties.get('test_apply_gerrit_ref'): if api.properties.get('test_apply_gerrit_ref'):
prop2arg = { prop2arg = {
...@@ -79,7 +80,8 @@ def RunSteps(api): ...@@ -79,7 +80,8 @@ def RunSteps(api):
suffix=suffix, suffix=suffix,
gerrit_no_reset=gerrit_no_reset, gerrit_no_reset=gerrit_no_reset,
gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref, gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref,
disable_syntax_validation=True) disable_syntax_validation=True,
manifest_name=manifest_name)
if patch: if patch:
api.bot_update.deapply_patch(bot_update_step) api.bot_update.deapply_patch(bot_update_step)
...@@ -89,6 +91,13 @@ def GenTests(api): ...@@ -89,6 +91,13 @@ def GenTests(api):
patch=False, patch=False,
revision='abc' revision='abc'
) )
yield api.test('with_manifest_name_no_patch') + api.properties(
manifest_name='checkout',
patch=False
)
yield api.test('with_manifest_name') + api.properties(
manifest_name='checkout'
)
yield api.test('buildbot') + api.properties( yield api.test('buildbot') + api.properties(
path_config='buildbot', path_config='buildbot',
patch=False, patch=False,
......
recipes/recipe_modules/bot_update/resources/bot_update.py
View file @ 037f9b80
...@@ -49,9 +49,6 @@ COMMIT_FOOTER_ENTRY_RE = re.compile(r'([^:]+):\s*(.*)') ...@@ -49,9 +49,6 @@ COMMIT_FOOTER_ENTRY_RE = re.compile(r'([^:]+):\s*(.*)')
COMMIT_POSITION_FOOTER_KEY = 'Cr-Commit-Position' COMMIT_POSITION_FOOTER_KEY = 'Cr-Commit-Position'
COMMIT_ORIGINAL_POSITION_FOOTER_KEY = 'Cr-Original-Commit-Position' COMMIT_ORIGINAL_POSITION_FOOTER_KEY = 'Cr-Original-Commit-Position'
# Regular expression to parse gclient's revinfo entries.
REVINFO_RE = re.compile(r'^([^:]+):\s+([^@]+)@(.+)$')
# Copied from scripts/recipes/chromium.py. # Copied from scripts/recipes/chromium.py.
GOT_REVISION_MAPPINGS = { GOT_REVISION_MAPPINGS = {
...@@ -364,18 +361,75 @@ def gclient_revinfo(): ...@@ -364,18 +361,75 @@ def gclient_revinfo():
return call_gclient('revinfo', '-a') or '' return call_gclient('revinfo', '-a') or ''
def create_manifest(): def normalize_git_url(url):
manifest = {} """Normalize a git url to be consistent.
output = gclient_revinfo()
for line in output.strip().splitlines(): This recognizes urls to the googlesoruce.com domain. It ensures that
match = REVINFO_RE.match(line.strip()) the url:
if match: * Do not end in .git
manifest[match.group(1)] = { * Do not contain /a/ in their path.
'repository': match.group(2), """
'revision': match.group(3), try:
p = urlparse.urlparse(url)
except Exception:
# Not a url, just return it back.
return url
if not p.netloc.endswith('.googlesource.com'):
# Not a googlesource.com URL, can't normalize this, just return as is.
return url
path = p.path
if path.startswith('/a'):
path = path[len('/a'):]
if path.endswith('.git'):
path = path[:-len('.git')]
return 'https://%s%s' % (p.netloc, path)
# TODO(hinoka): Include patch revision.
def create_manifest(gclient_output, patch_root, gerrit_ref):
"""Return the JSONPB equivilent of the source manifest proto.
The source manifest proto is defined here:
https://chromium.googlesource.com/infra/luci/recipes-py/+/master/recipe_engine/source_manifest.proto
This is based off of:
* The gclient_output (from calling gclient.py --output-json) which contains
the directory -> repo:revision mapping.
* Gerrit Patch info which contains info about patched revisions.
We normalize the URLs such that if they are googlesource.com urls, they:
"""
manifest = {
'version': 0, # Currently the only valid version is 0.
}
dirs = {}
if patch_root:
patch_root = patch_root.strip('/') # Normalize directory names.
for directory, info in gclient_output.get('solutions', {}).iteritems():
directory = directory.strip('/') # Normalize the directory name.
# There are two places to the the revision from, we do it in this order:
# 1. In the "revision" field
# 2. At the end of the URL, after @
repo = ''
revision = info.get('revision', '')
# The format of the url is "https://repo.url/blah.git@abcdefabcdef" or
# just "https://repo.url/blah.git"
url_split = info.get('url', '').split('@')
if not revision and len(url_split) == 2:
revision = url_split[1]
if url_split:
repo = normalize_git_url(url_split[0])
if repo:
dirs[directory] = {
'git_checkout': {
'repo_url': repo,
'revision': revision,
}
} }
else: if patch_root == directory:
print "WARNING: Couldn't match revinfo line:\n%s" % line dirs[directory]['git_checkout']['patch_fetch_ref'] = gerrit_ref
manifest['directories'] = dirs
return manifest return manifest
...@@ -1104,7 +1158,8 @@ def checkout(options, git_slns, specs, revisions, step_text, shallow): ...@@ -1104,7 +1158,8 @@ def checkout(options, git_slns, specs, revisions, step_text, shallow):
step_text=step_text, step_text=step_text,
fixed_revisions=revisions, fixed_revisions=revisions,
properties=got_revisions, properties=got_revisions,
manifest=create_manifest()) manifest=create_manifest(
gclient_output, options.patch_root, options.gerrit_ref))
def print_debug_info(): def print_debug_info():
......
tests/bot_update_coverage_test.py
View file @ 037f9b80
...@@ -245,6 +245,38 @@ class BotUpdateUnittests(unittest.TestCase): ...@@ -245,6 +245,38 @@ class BotUpdateUnittests(unittest.TestCase):
setattr(os, 'remove', old_os_remove) setattr(os, 'remove', old_os_remove)
self.assertTrue(os.path.join(path, lockfile) in removed) self.assertTrue(os.path.join(path, lockfile) in removed)
def testGenerateManifestsBasic(self):
gclient_output = {
'solutions': {
'breakpad/': {
'revision': None,
'scm': None,
'url': ('https://chromium.googlesource.com/breakpad/breakpad.git' +
'@5f638d532312685548d5033618c8a36f73302d0a')
},
"src/": {
'revision': 'f671d3baeb64d9dba628ad582e867cf1aebc0207',
'scm': None,
'url': 'https://chromium.googlesource.com/a/chromium/src.git'
},
}
}
out = bot_update.create_manifest(gclient_output, None, None)
self.assertEquals(len(out['directories']), 2)
print out
self.assertEquals(
out['directories']['src']['git_checkout']['revision'],
'f671d3baeb64d9dba628ad582e867cf1aebc0207')
self.assertEquals(
out['directories']['src']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/chromium/src')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['revision'],
'5f638d532312685548d5033618c8a36f73302d0a')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/breakpad/breakpad')
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment