aiden
/
depot_tools
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

bot_update recipe: Upload source manifest

Browse Source 
Bug: 772529
Change-Id: I3f833740ddfbc71be42b140c99dcd4bbfeed54fe
Reviewed-on: https://chromium-review.googlesource.com/706071
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
changes/71/706071/17
Ryan Tseng 8 years ago committed by Commit Bot
parent e6e0641b15
commit 037f9b80bb
8 changed files with 363 additions and 22 deletions
Show Stats Download Patch File Download Diff File

11
recipes/README.recipes.md
Unescape Escape View File

@ -38,7 +38,7 @@
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [rietveld](#recipe_modules-rietveld), [tryserver](#recipe_modules-tryserver), [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/source\_manifest][recipe_engine/recipe_modules/source_manifest], [recipe\_engine/step][recipe_engine/recipe_modules/step]
Recipe module to ensure a checkout is consistent on a bot.
@ -50,12 +50,12 @@ Wrapper for easy calling of bot_update.
&mdash; **def [apply\_gerrit\_ref](/recipes/recipe_modules/bot_update/api.py#49)(self, root, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, gerrit_repo=None, gerrit_ref=None, step_name='apply_gerrit', \*\*kwargs):**
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#434)(self, bot_update_step):**
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#445)(self, bot_update_step):**
Deapplies a patch, taking care of DEPS and solution revisions properly.
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, \*\*kwargs):**
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#71)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, no_shallow=False, with_branch_heads=False, with_tags=False, refs=None, patch_oauth2=False, oauth2_json=False, use_site_config_creds=True, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, manifest_name=None, \*\*kwargs):**
Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot
@ -71,8 +71,10 @@ Args:
disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions,
such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#411)(self, project_name, gclient_config=None):**
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#422)(self, project_name, gclient_config=None):**
Returns all property names used for storing the checked-out revision of
a given project.
@ -839,6 +841,7 @@ Move things around in a loop!
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-properties
[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-python
[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-raw_io
[recipe_engine/recipe_modules/source_manifest]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-source_manifest
[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-step
[recipe_engine/recipe_modules/url]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/README.recipes.md#recipe_modules-url
[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/c07de907b2223efcacbf45d7b793fc0d8a964f31/recipe_engine/recipe_api.py#992

1
recipes/recipe_modules/bot_update/__init__.py
Unescape Escape View File

@ -9,6 +9,7 @@ DEPS = [
'recipe_engine/properties',
'recipe_engine/python',
'recipe_engine/raw_io',
'recipe_engine/source_manifest',
'recipe_engine/step',
'rietveld',
'tryserver',

15
recipes/recipe_modules/bot_update/api.py
Unescape Escape View File

@ -77,7 +77,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
root_solution_revision=None, rietveld=None, issue=None,
patchset=None, gerrit_no_reset=False,
gerrit_no_rebase_patch_ref=False,
disable_syntax_validation=False, **kwargs):
disable_syntax_validation=False, manifest_name=None,
**kwargs):
"""
Args:
use_site_config_creds: If the oauth2 credentials are in the buildbot
@ -93,6 +94,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
disable_syntax_validation: (legacy) Disables syntax validation for DEPS.
Needed as migration paths for recipes dealing with older revisions,
such as bisect.
manifest_name: The name of the manifest to upload to LogDog. This must
be unique for the whole build.
"""
refs = refs or []
# We can re-use the gclient spec from the gclient module, since all the
@ -261,7 +264,6 @@ class BotUpdateApi(recipe_api.RecipeApi):
root, first_sln, reverse_rev_map, self._fail_patch,
fixed_revisions=fixed_revisions)
# Add suffixes to the step name, if specified.
name = 'bot_update'
if not patch:
name += ' (without patch)'
@ -296,6 +298,15 @@ class BotUpdateApi(recipe_api.RecipeApi):
step_text = result['step_text']
step_result.presentation.step_text = step_text
# Export the step results as a Source Manifest to LogDog.
if manifest_name:
if not patch:
# The param "patched" is purely cosmetic to mean "if false, this
# bot_update run exists purely to unpatch an existing patch".
manifest_name += '_unpatched'
self.m.source_manifest.set_json_manifest(
manifest_name, result.get('manifest', {}))
# Set the "checkout" path for the main solution.
# This is used by the Chromium module to figure out where to look for
# the checkout.

148
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name.json
Unescape Escape View File

@ -0,0 +1,148 @@
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]

82
recipes/recipe_modules/bot_update/examples/full.expected/with_manifest_name_no_patch.json
Unescape Escape View File

@ -0,0 +1,82 @@
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[GIT_CACHE]'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[GIT_CACHE]",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@HEAD",
"--disable-syntax-validation"
],
"env_prefixes": {
"PATH": [
"RECIPE_PACKAGE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"name": "$result",
"recipe_result": null,
"status_code": 0
}
]

11
recipes/recipe_modules/bot_update/examples/full.py
Unescape Escape View File

@ -49,6 +49,7 @@ def RunSteps(api):
gerrit_no_reset = True if api.properties.get('gerrit_no_reset') else False
gerrit_no_rebase_patch_ref = bool(
api.properties.get('gerrit_no_rebase_patch_ref'))
manifest_name = api.properties.get('manifest_name')
if api.properties.get('test_apply_gerrit_ref'):
prop2arg = {
@ -79,7 +80,8 @@ def RunSteps(api):
suffix=suffix,
gerrit_no_reset=gerrit_no_reset,
gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref,
disable_syntax_validation=True)
disable_syntax_validation=True,
manifest_name=manifest_name)
if patch:
api.bot_update.deapply_patch(bot_update_step)
@ -89,6 +91,13 @@ def GenTests(api):
patch=False,
revision='abc'
)
yield api.test('with_manifest_name_no_patch') + api.properties(
manifest_name='checkout',
patch=False
)
yield api.test('with_manifest_name') + api.properties(
manifest_name='checkout'
)
yield api.test('buildbot') + api.properties(
path_config='buildbot',
patch=False,

85
recipes/recipe_modules/bot_update/resources/bot_update.py
Unescape Escape View File

@ -49,9 +49,6 @@ COMMIT_FOOTER_ENTRY_RE = re.compile(r'([^:]+):\s*(.*)')
COMMIT_POSITION_FOOTER_KEY = 'Cr-Commit-Position'
COMMIT_ORIGINAL_POSITION_FOOTER_KEY = 'Cr-Original-Commit-Position'
# Regular expression to parse gclient's revinfo entries.
REVINFO_RE = re.compile(r'^([^:]+):\s+([^@]+)@(.+)$')
# Copied from scripts/recipes/chromium.py.
GOT_REVISION_MAPPINGS = {
@ -364,18 +361,75 @@ def gclient_revinfo():
return call_gclient('revinfo', '-a') or ''
def create_manifest():
manifest = {}
output = gclient_revinfo()
for line in output.strip().splitlines():
match = REVINFO_RE.match(line.strip())
if match:
manifest[match.group(1)] = {
'repository': match.group(2),
'revision': match.group(3),
def normalize_git_url(url):
"""Normalize a git url to be consistent.
This recognizes urls to the googlesoruce.com domain. It ensures that
the url:
* Do not end in .git
* Do not contain /a/ in their path.
"""
try:
p = urlparse.urlparse(url)
except Exception:
# Not a url, just return it back.
return url
if not p.netloc.endswith('.googlesource.com'):
# Not a googlesource.com URL, can't normalize this, just return as is.
return url
path = p.path
if path.startswith('/a'):
path = path[len('/a'):]
if path.endswith('.git'):
path = path[:-len('.git')]
return 'https://%s%s' % (p.netloc, path)
# TODO(hinoka): Include patch revision.
def create_manifest(gclient_output, patch_root, gerrit_ref):
"""Return the JSONPB equivilent of the source manifest proto.
The source manifest proto is defined here:
https://chromium.googlesource.com/infra/luci/recipes-py/+/master/recipe_engine/source_manifest.proto
This is based off of:
* The gclient_output (from calling gclient.py --output-json) which contains
the directory -> repo:revision mapping.
* Gerrit Patch info which contains info about patched revisions.
We normalize the URLs such that if they are googlesource.com urls, they:
"""
manifest = {
'version': 0, # Currently the only valid version is 0.
}
else:
print "WARNING: Couldn't match revinfo line:\n%s" % line
dirs = {}
if patch_root:
patch_root = patch_root.strip('/') # Normalize directory names.
for directory, info in gclient_output.get('solutions', {}).iteritems():
directory = directory.strip('/') # Normalize the directory name.
# There are two places to the the revision from, we do it in this order:
# 1. In the "revision" field
# 2. At the end of the URL, after @
repo = ''
revision = info.get('revision', '')
# The format of the url is "https://repo.url/blah.git@abcdefabcdef" or
# just "https://repo.url/blah.git"
url_split = info.get('url', '').split('@')
if not revision and len(url_split) == 2:
revision = url_split[1]
if url_split:
repo = normalize_git_url(url_split[0])
if repo:
dirs[directory] = {
'git_checkout': {
'repo_url': repo,
'revision': revision,
}
}
if patch_root == directory:
dirs[directory]['git_checkout']['patch_fetch_ref'] = gerrit_ref
manifest['directories'] = dirs
return manifest
@ -1104,7 +1158,8 @@ def checkout(options, git_slns, specs, revisions, step_text, shallow):
step_text=step_text,
fixed_revisions=revisions,
properties=got_revisions,
manifest=create_manifest())
manifest=create_manifest(
gclient_output, options.patch_root, options.gerrit_ref))
def print_debug_info():

32
tests/bot_update_coverage_test.py
Unescape Escape View File

@ -245,6 +245,38 @@ class BotUpdateUnittests(unittest.TestCase):
setattr(os, 'remove', old_os_remove)
self.assertTrue(os.path.join(path, lockfile) in removed)
def testGenerateManifestsBasic(self):
gclient_output = {
'solutions': {
'breakpad/': {
'revision': None,
'scm': None,
'url': ('https://chromium.googlesource.com/breakpad/breakpad.git' +
'@5f638d532312685548d5033618c8a36f73302d0a')
},
"src/": {
'revision': 'f671d3baeb64d9dba628ad582e867cf1aebc0207',
'scm': None,
'url': 'https://chromium.googlesource.com/a/chromium/src.git'
},
}
}
out = bot_update.create_manifest(gclient_output, None, None)
self.assertEquals(len(out['directories']), 2)
print out
self.assertEquals(
out['directories']['src']['git_checkout']['revision'],
'f671d3baeb64d9dba628ad582e867cf1aebc0207')
self.assertEquals(
out['directories']['src']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/chromium/src')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['revision'],
'5f638d532312685548d5033618c8a36f73302d0a')
self.assertEquals(
out['directories']['breakpad']['git_checkout']['repo_url'],
'https://chromium.googlesource.com/breakpad/breakpad')
if __name__ == '__main__':
unittest.main()

Write Preview
Loading…
Cancel
Save