Modify bot_update to support setting blamelist pins.

This adds a parameter to bot_update.ensure_checkout that will cause
blamelist pins to be set for all of the repos whose revision is not
determined by DEPS files: the repo for the gitiles input, the repo for
each solution in the gclient config and any repos with the revision set
by the revisions field of the gclient config.

Bug: 1047893
Change-Id: Ie3eee41dbf15be014185b6f255ee3d4cfd40b4e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2293514
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
changes/14/2293514/6
Garrett Beaty 5 years ago committed by LUCI CQ
parent c8c88f9c55
commit 6081b23b29

@ -48,7 +48,7 @@
### *recipe_modules* / [bot\_update](/recipes/recipe_modules/bot_update)
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/commit\_position][recipe_engine/recipe_modules/commit_position], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
[DEPS](/recipes/recipe_modules/bot_update/__init__.py#1): [depot\_tools](#recipe_modules-depot_tools), [gclient](#recipe_modules-gclient), [gerrit](#recipe_modules-gerrit), [gitiles](#recipe_modules-gitiles), [tryserver](#recipe_modules-tryserver), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/commit\_position][recipe_engine/recipe_modules/commit_position], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/cq][recipe_engine/recipe_modules/cq], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/milo][recipe_engine/recipe_modules/milo], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/runtime][recipe_engine/recipe_modules/runtime], [recipe\_engine/step][recipe_engine/recipe_modules/step]
Recipe module to ensure a checkout is consistent on a bot.
@ -58,12 +58,12 @@ Recipe module to ensure a checkout is consistent on a bot.
Wrapper for easy calling of bot_update.
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#500)(self, bot_update_step):**
&mdash; **def [deapply\_patch](/recipes/recipe_modules/bot_update/api.py#515)(self, bot_update_step):**
Deapplies a patch, taking care of DEPS and solution revisions properly.
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#76)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, with_branch_heads=False, with_tags=False, no_fetch_tags=False, refs=None, patch_oauth2=None, oauth2_json=None, use_site_config_creds=None, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, patch_refs=None, ignore_input_commit=False, set_output_commit=False, step_test_data=None, \*\*kwargs):**
&mdash; **def [ensure\_checkout](/recipes/recipe_modules/bot_update/api.py#76)(self, gclient_config=None, suffix=None, patch=True, update_presentation=True, patch_root=None, with_branch_heads=False, with_tags=False, no_fetch_tags=False, refs=None, patch_oauth2=None, oauth2_json=None, use_site_config_creds=None, clobber=False, root_solution_revision=None, rietveld=None, issue=None, patchset=None, gerrit_no_reset=False, gerrit_no_rebase_patch_ref=False, disable_syntax_validation=False, patch_refs=None, ignore_input_commit=False, add_blamelists=False, set_output_commit=False, step_test_data=None, \*\*kwargs):**
Args:
gclient_config: The gclient configuration to use when running bot_update.
@ -77,6 +77,8 @@ Args:
such as bisect.
ignore_input_commit: if True, ignore api.buildbucket.gitiles_commit.
Exists for historical reasons. Please do not use.
add_blamelists: if True, add blamelist pins for all of the repos that had
revisions specified in the gclient config.
set_output_commit: if True, mark the checked out commit as the
primary output commit of this build, i.e. call
api.buildbucket.set_output_gitiles_commit.
@ -87,7 +89,7 @@ Args:
step_test_data: a null function that returns test bot_update.py output.
Use test_api.output_json to generate test data.
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#477)(self, project_name, gclient_config=None):**
&mdash; **def [get\_project\_revision\_properties](/recipes/recipe_modules/bot_update/api.py#492)(self, project_name, gclient_config=None):**
Returns all property names used for storing the checked-out revision of
a given project.
@ -105,7 +107,7 @@ Returns (list of str): All properties that'll hold the checked-out revision
&emsp; **@property**<br>&mdash; **def [last\_returned\_properties](/recipes/recipe_modules/bot_update/api.py#44)(self):**
&mdash; **def [resolve\_fixed\_revision](/recipes/recipe_modules/bot_update/api.py#427)(self, bot_update_json, name):**
&mdash; **def [resolve\_fixed\_revision](/recipes/recipe_modules/bot_update/api.py#442)(self, bot_update_json, name):**
Set a fixed revision for a single dependency using project revision
properties.
@ -1067,6 +1069,7 @@ Move things around in a loop!
[recipe_engine/recipe_modules/cq]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-cq
[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-file
[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-json
[recipe_engine/recipe_modules/milo]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-milo
[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-path
[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-platform
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/79f951ca7bfd9a119dbe0cb27a51bf78493836bd/README.recipes.md#recipe_modules-properties

@ -8,6 +8,7 @@ DEPS = [
'recipe_engine/commit_position',
'recipe_engine/cq',
'recipe_engine/json',
'recipe_engine/milo',
'recipe_engine/path',
'recipe_engine/platform',
'recipe_engine/properties',

@ -96,6 +96,7 @@ class BotUpdateApi(recipe_api.RecipeApi):
disable_syntax_validation=False,
patch_refs=None,
ignore_input_commit=False,
add_blamelists=False,
set_output_commit=False,
step_test_data=None,
**kwargs):
@ -112,6 +113,8 @@ class BotUpdateApi(recipe_api.RecipeApi):
such as bisect.
ignore_input_commit: if True, ignore api.buildbucket.gitiles_commit.
Exists for historical reasons. Please do not use.
add_blamelists: if True, add blamelist pins for all of the repos that had
revisions specified in the gclient config.
set_output_commit: if True, mark the checked out commit as the
primary output commit of this build, i.e. call
api.buildbucket.set_output_gitiles_commit.
@ -314,6 +317,18 @@ class BotUpdateApi(recipe_api.RecipeApi):
step_text = result['step_text']
step_result.presentation.step_text = step_text
if add_blamelists:
blamelist_pins = []
for name in revisions:
m = result['manifest'][name]
pin = {'id': m['revision']}
pin['host'], pin['project'] = (
self.m.gitiles.parse_repo_url(m['repository']))
blamelist_pins.append(pin)
result.blamelist_pins = blamelist_pins
self.m.milo.show_blamelist_for(blamelist_pins)
# Set output commit of the build.
if set_output_commit:
# As of April 2019, got_revision describes the output commit,

@ -0,0 +1,127 @@
[
{
"cmd": [
"python",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[CACHE]/git'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'https://chromium.googlesource.com/chromium/src.git'}]",
"--revision_mapping_file",
"{\"got_angle_revision\": \"src/third_party/angle\", \"got_cr_revision\": \"src\", \"got_revision\": \"src\", \"got_v8_revision\": \"src/v8\"}",
"--git-cache-dir",
"[CACHE]/git",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@2d72510e447ab60a9728aeea2362d8be2cbd7789",
"--revision",
"src/v8@HEAD",
"--refs",
"refs/heads/master",
"--disable-syntax-validation"
],
"env": {
"GIT_HTTP_LOW_SPEED_LIMIT": "102400",
"GIT_HTTP_LOW_SPEED_TIME": "300"
},
"env_suffixes": {
"DEPOT_TOOLS_UPDATE": [
"0"
],
"PATH": [
"RECIPE_REPO[depot_tools]"
]
},
"infra_step": true,
"name": "bot_update (without patch)",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_angle_revision_cp\": \"refs/heads/master@{#297276}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_cr_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/master@{#170242}\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision\": \"801ada225ddc271c132c3a35f03975671d43e399\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_v8_revision_cp\": \"refs/heads/master@{#43426}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"source_manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"directories\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"git_checkout\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repo_url\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/third_party/angle\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"git_checkout\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repo_url\": \"https://fake.org/src/third_party/angle.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"src/v8\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"git_checkout\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repo_url\": \"https://fake.org/src/v8.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"version\": 0@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision@\"fac9503c46405f77757b9a728eb85b8d7bc6080c\"@@@",
"@@@SET_BUILD_PROPERTY@got_angle_revision_cp@\"refs/heads/master@{#297276}\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision@\"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@SET_BUILD_PROPERTY@got_cr_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/master@{#170242}\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision@\"801ada225ddc271c132c3a35f03975671d43e399\"@@@",
"@@@SET_BUILD_PROPERTY@got_v8_revision_cp@\"refs/heads/master@{#43426}\"@@@"
]
},
{
"cmd": [],
"name": "set blamelist pins",
"~followup_annotations": [
"@@@SET_BUILD_PROPERTY@$recipe_engine/milo/blamelist_pins@[{\"host\": \"fake.org\", \"id\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", \"project\": \"src\"}, {\"host\": \"fake.org\", \"id\": \"801ada225ddc271c132c3a35f03975671d43e399\", \"project\": \"src/v8\"}]@@@"
]
},
{
"cmd": [],
"name": "set_output_gitiles_commit",
"~followup_annotations": [
"@@@SET_BUILD_PROPERTY@$recipe_engine/buildbucket/output_gitiles_commit@{\"host\": \"fake.org\", \"id\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", \"position\": 170242, \"project\": \"src\", \"ref\": \"refs/heads/master\"}@@@"
]
},
{
"name": "$result"
}
]

@ -15,8 +15,6 @@
"--output_json",
"/path/to/tmp/json",
"--revision",
"got_revision@src",
"--revision",
"src@HEAD",
"--refs",
"refs/heads/master",

@ -15,8 +15,6 @@
"--output_json",
"/path/to/tmp/json",
"--revision",
"got_revision@src",
"--revision",
"src@refs/branch-heads/x",
"--refs",
"refs/branch-heads/x",

@ -15,8 +15,6 @@
"--output_json",
"/path/to/tmp/json",
"--revision",
"got_revision@src",
"--revision",
"src@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"--refs",
"refs/heads/master",

@ -15,8 +15,6 @@
"--output_json",
"/path/to/tmp/json",
"--revision",
"got_revision@src",
"--revision",
"src@refs/heads/master",
"--refs",
"refs/heads/master",

@ -56,6 +56,7 @@ def RunSteps(api):
gerrit_no_rebase_patch_ref = bool(
api.properties.get('gerrit_no_rebase_patch_ref'))
patch_refs = api.properties.get('patch_refs')
add_blamelists = api.properties.get('add_blamelists', False)
set_output_commit = api.properties.get('set_output_commit', True)
step_test_data = None
@ -74,6 +75,7 @@ def RunSteps(api):
gerrit_no_rebase_patch_ref=gerrit_no_rebase_patch_ref,
disable_syntax_validation=True,
patch_refs=patch_refs,
add_blamelists=add_blamelists,
set_output_commit=set_output_commit,
step_test_data=step_test_data,
)
@ -231,11 +233,19 @@ def GenTests(api):
)
)
yield (
api.test('add_blamelists') +
ci_build() +
api.properties(
add_blamelists=True,
revisions={'src/v8': 'HEAD'},
)
)
yield (
api.test('no_cp_checkout_a_specific_commit') +
ci_build(revision='a' * 40) +
api.properties(
revisions={'got_revision': 'src'},
bot_update_output={
'properties': {
'got_revision': 'a' * 40,
@ -254,7 +264,6 @@ def GenTests(api):
api.test('no_cp_checkout_master') +
ci_build(revision='') +
api.properties(
revisions={'got_revision': 'src'},
bot_update_output={
'properties': {
'got_revision': 'a' * 40,
@ -273,7 +282,6 @@ def GenTests(api):
api.test('no_cp_checkout_a_branch_head') +
ci_build(revision='', git_ref='refs/branch-heads/x') +
api.properties(
revisions={'got_revision': 'src'},
bot_update_output={
'properties': {
'got_revision': 'a' * 40,
@ -292,7 +300,6 @@ def GenTests(api):
api.test('no_cp_checkout_HEAD') +
ci_build(revision='HEAD') +
api.properties(
revisions={'got_revision': 'src'},
bot_update_output={
'properties': {
'got_revision': 'a' * 40,

Loading…
Cancel
Save