By default during tests, the _get_footers_step call will return None for
the returned footers because it does not set any test data. This does
not match the actual behavior of the recipe because git_footers.py
returns an empty json object if the commit message has no footers. This
change updates it to return an empty dict instead of None and checks for
the cached footers value to be None instead of any non-true value so
that a commit message without footers doesn't get parsed multiple times.
Change-Id: I716a27e964eb92de138228df5cc0876322d82823
Recipe-Manual-Change: build
Recipe-Nontrivial-Roll: build_limited
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4492269
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
https://crrev.com/c/4420526 put a 10s timeout on all calls made via
this script.
But it appears there are a variety of endpoints that take more than
10s, the latest being "SubmitChange", eg:
https://ci.chromium.org/ui/p/chromeos/builders/release/release-R113-15393.B-orchestrator/25/overview
So instead of applying the timeout to everything, and then opting
certain long-running calls out of it, this just opts-in the
"GetChanges" call to the timeout. This should cover the failure mode
that was originally solved for chrome's bots. And also bump the
timeout to 30s since we don't trust all get-changes queries to
reliably finish in under 10s.
This also bumps the step level timeout for the query in tryserver
recipe_mod to 8 min since there could be ~5min worth of sleeping +
150s worth of waiting+timing-out. So worst case the step will now
take 8min to fully exhaust all timeouts/sleeps.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: b/278083716
Change-Id: Ib366e004e0bb07297ba732590d488cae779e38ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4426524
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
When gerrit's availability drops, chrome's builds see an excessive
amount of failures, eg:
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359780/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359594/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359564/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359630/overview
Seemingly all failures occur in either the `gerrit fetch current CL
info` step or the `gerrit changes` step. Both steps have a 60s timeout
and shell out to depot_tools' gerrit_client.py. That script essentially
makes a single http call to gerrit. That request has no configured
timeout. So when gerrit's MIA and the call hangs indefinitely, so too
will the step hang. 60s after that, the step timeout is reached, and the
entire build crashes with an infra-failure.
However, one single retry has been shown to sufficiently work around
at least one instance of that failure:
dea9a6eca26f690ce10a959701a8fb51ad/+/build.proto
So this incorporates timeouts into the requests made by
gerrit_util.py. Each request is given a 10s timeout, which should be
enough for most/all gerrit calls. (Both steps have a p90 of less than
1sec.) When a timeout is reached, the script will automatically retry
up to 4 times.
This also bumps the timeouts of the step calls to gerrit_client.py to
6min since the script can now take up to 5 minutes to fail, ie:
the sequence of sleeps is roughly 10s, 20s, 40s, 80s, 160s, which is
about 5min. So a 6min timeout should cover all those retries.
This also passes in "--verbose" to all step calls to this script, so
the logging that prints out the retry + sleep log lines will be
visible in build logs.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: 1432638
Change-Id: I9dc47f4beeda3783ae4f9152bd29ee441ac3e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4420526
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Removes a TODO to remove the patch_text argument and its uses from
get_footers and related methods. The TODO indicated that it was
only used in example code, and thus should be handled another way.
However, it turns out that there is at least one legitimate use of
thus functionality downstream to retrieve footers from CL
descriptions earlier revisions.
Fixed: 1179039
Change-Id: I79996f6053a6769043943de39735b2b5d7ee02ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4283342
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Auto-Submit: Brian Sheedy <bsheedy@chromium.org>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
The get_files_affected_by_patch method on the test API will return a
TestData object that determines the output of the "git diff" call. a
Test was added specifically for the get_files_affected_by_patch API
method to ensure that the test API operates as expected.
Change-Id: Ia1846700461757a72ed518894a5a38a8ca54fcc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3422006
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
The tryserver.require_is_tryserver method will create an infra-failing
step that raises an exception if there is no CL associated with the
build. In the case of an LED task, it will only fail so as not to cause
the removal of the builder cache in the case of a minor user error.
Change-Id: I05d3a86cc727577f067fb3eae49e46398345f672
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413971
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Allows for quick retrieval of host and project without having to parse
gerrit_change_repo_url or hard coding.
Bug:
Change-Id: I43f6e0bfda4b54d742bfa7ea20ee781162796608
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2971110
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Gonna add a small warning in chromium's recipe when this footer is
used. Add it here so we can refer to the footer as a constant in
tryserver recipe_mod.
Bug: None
Change-Id: Ie4f9d1d208c0dad8d0df8f07b74e761da1051a1c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2864005
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
In crbug.com/1196159, a build got stuck at "gerrit changes" step for 3
hours. It would be good to add a timeout so that it fails early.
Also adjust the timeout for get_changes to 60s, from 600s
Bug: 1196159
Change-Id: Id2327ab40dd2fd8d95aaad6bb724f7ed06e6c244
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2811899
Commit-Queue: Haiyang Pan <hypan@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Includes all footers used by the recipe. Other footers used only by presubmit
hooks not included.
Bug: chromium:1079219
Change-Id: I9c351bda158bc7c1e64a8254feb7055dc31fb6db
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2707709
Auto-Submit: Jacob Kopczynski <jkop@chromium.org>
Commit-Queue: Jacob Kopczynski <jkop@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
It will be easier to support centralized management of what footers exist if we
require all callers to specifically declare which footers they want to check.
This is a partial step towards allowing that without increasing the number of
queries required. It introduced cache staleness as a potential issue, but should
not matter in practice.
Bug: chromium:1079219
Change-Id: I678fd366202bac5a4efa4258ffe02e70c64589bb
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: chromiumos
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2664790
Commit-Queue: Jacob Kopczynski <jkop@chromium.org>
Auto-Submit: Jacob Kopczynski <jkop@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
To enable aggregate analysis on changed files in BigQuery, conditionally
report them via an output property.
This CL causes a non-trivial roll only because of the sorting.
Recipe-Nontrivial-Roll: build
Bug: 1151655
Change-Id: Ie7a6c622196143c1aef07af9a9a356fb4202ea56
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2553349
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
This reverts commit 4447336100.
Reason for revert: Breaks infra repo. See crrev.com/c/2548285
Original change's description:
> Improve the setup logic for tryserver
>
> If the given change does not have a valid gerrit host name, stop
> initializing the tryserver module.
> Also polished the variable name and docstr added in crrev/c/2528670.
>
> Recipe-Nontrivial-Roll: infra
> Bug:1146487
> Change-Id: I38193b25d37c724fd41d898eb020237fbf7cecdf
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2544000
> Commit-Queue: Xinan Lin <linxinan@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
TBR=nodir@chromium.org,tandrii@google.com,recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com,linxinan@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com
Change-Id: I3871866454adab816ab5549f8273fe16af2d9adb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1146487
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2548409
Reviewed-by: Xinan Lin <linxinan@chromium.org>
Commit-Queue: Xinan Lin <linxinan@chromium.org>
If the given change does not have a valid gerrit host name, stop
initializing the tryserver module.
Also polished the variable name and docstr added in crrev/c/2528670.
Recipe-Nontrivial-Roll: infra
Bug:1146487
Change-Id: I38193b25d37c724fd41d898eb020237fbf7cecdf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2544000
Commit-Queue: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Currently bot_update and tryserver crashes at initialization step if
the input contains multiple gerrit changes. This is not friendly for
CrOS CLs, e.g. crrev.com/c/2536761.
This CL moves the “crash” out of initialization, thus the consumer
of these modules could control which gerrit change to pass to bot_update
and tryserver.
This CL does NOT enable the modules here to support multiple patchset,
but allows consumers to choose one.
Bug:1146487
Change-Id: Iec46a8041189912e34da9bc5e8ff4611123ec9e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2528670
Commit-Queue: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
To sanely support concurrency, I need to make `api.step.nest` strictly
scoped; i.e. once you leave the nest context it closes all the steps
within that context.
This changes some wild 'active_step' uses to generate a step for the
module's own purpose.
R=martiniss@chromium.org, nodir@chromium.org, tandrii@chromium.org
Recipe-Nontrivial-Roll: build
Bug: 910369
Change-Id: I19aef31118faf9d3d078a2c5a0ebd48a4bfebd43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1630044
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
The function can be called by chromium_test to set a property 'do_not_retry'
which will be propagated into buildbucket output.
Change-Id: I32d8ea925b7cb98d9b25d24226686e116c17801c
Bug: 910193
Reviewed-on: https://chromium-review.googlesource.com/c/1351542
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
It is not used. get_gerrit_patch_root is used instead.
Bug: 694348
Change-Id: Icc717e38cbcaa8050c8ce8a364202b81d7f9cf2c
Reviewed-on: https://chromium-review.googlesource.com/1231282
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Add tryserver.api.gerrit_change_info that returns detailed info about the
current Gerrit CL. It makes a Gerrit RPC in the background on the first read.
This step will be present in all tryjobs.
This mechanism can be used for extra details about the current CL.
This CL starts to use this in bot_update for gerrit_ref, and stops using
patch_ref property, and also update bot_update to use when the destination
branch is needed.
Also this CL removes no longer used properties in bot_update,
namely most of patch_* properties.
build.git manual roll: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1217626
Recipe-Manual-Change: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: release_scripts
R=tandrii@chromium.org
Bug: 694348
Change-Id: I945d061b74278c3d1d227578abe16a64cdaca647
Reviewed-on: https://chromium-review.googlesource.com/1213557
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
tryserver module's purpose is to provide high level API for patches.
Update it to use buildbucket.build.input.gerrit_changes[0].
Add helper properties for gerrit repo URL and gerrit ref.
This will be used by both bot_update and gclient.
R=hinoka@chromium.org, tandrii@chromium.org
Bug: 694348
Change-Id: I7eb416e75737310404c4354fa680fd3499eddaa7
Reviewed-on: https://chromium-review.googlesource.com/1211653
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This introduces a pragma: no cover, which is unfortunate.
But this is a prerequisite for removing rietveld support
entirely from the recipe engine without having to do
manual recipe rolls. The entire block of uncovered code
will be removed shortly.
Bug: 770408
Change-Id: I37319f3ae70413adcda655c443325bccffe695d8
Reviewed-on: https://chromium-review.googlesource.com/834455
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
This is a reland of 9219d35688
The original was reverted due to a typo (core,quotePath instead
of core.quotePath). This version is fixed.
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
>
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I28d2260948aaf63bd865888c2f60e4cdee9aea48
Reviewed-on: https://chromium-review.googlesource.com/822990
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This reverts commit 9219d35688.
Reason for revert: unfortunately this says "core,quotePath" and since it includes recipe changes, we need something that the roller can munch on :(
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
TBR=maruel@chromium.org,agable@chromium.org
Change-Id: I226388f19024403240a1443eb2b878b9293220e1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 792302
Reviewed-on: https://chromium-review.googlesource.com/821671
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
This prevents git from putting quotes around some file names
(those that have astral-plane characters) and not around others.
R=maruel
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
Reviewed-on: https://chromium-review.googlesource.com/815454
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Rob Mohr
Robert Iannucci
Garrett Beaty
Ariel Zhang
Ben Pastene
Brian Sheedy
Takuto Ikuta
Andrew Grieve
Gavin Mak
Josip Sokcevic
Stephanie Kim
Haiyang Pan
Greg Guterman
Jacob Kopczynski
Nodir Turakulov
Xinan Lin
Quinten Yearsley
Andrii Shyshkalov
Marc-Antoine Ruel
Erik Staab
John Budorick
erikchen
Nodir Turakulov
Aaron Gable
Daniel Jacques
Paweł Hajdan, Jr