python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I479de09a0c34b438aced35e4ced58a5972108132
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4808518
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
The "upload traces" step is skipped if bot_update times out and is in
the grace_period. Reserve 1 minute to upload traces.
Bug: 1351547, 1456840
Change-Id: Idb191735a07d245575ff4c79375a976750522fa5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4651461
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Since python3.3, the platform value has been changed from linux2 or
linux3 to linux. The bug has stopped bot_update script from emitting
proces tree periodically that will be helpful when debugging.
R=sokcevic
Bug: 1455926
Change-Id: Id6ab4ddf46eaa322a8b5470baa7ba6cc656a6bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4654290
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
The blamelist_pins attribute is being set on the json output, which is a
dictionary. This only works right now because json.output currently uses
OrderedDict as the dict type. With removal of support for python2, dict
could be used since its iteration order is insertion order in python3.
This change removes setting the blamelist_pins attribute since its not
used anywhere and it will allow json.output to use plain dict.
Change-Id: Ie19b87d61903774454df2d395acf695eaa39b201
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4617204
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
This change adds git cache's epoch timestamp to bot_update's output.properties. It achieves this by adding a marker file to the cache directory which stores the epoch time.
Design: go/query-bot-update-cache-age
Bug: 1448769
Change-Id: I610c6bca5ced7e2a0fe3ee8570decf0f135fe54c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4563726
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
https://crrev.com/c/4420526 put a 10s timeout on all calls made via
this script.
But it appears there are a variety of endpoints that take more than
10s, the latest being "SubmitChange", eg:
https://ci.chromium.org/ui/p/chromeos/builders/release/release-R113-15393.B-orchestrator/25/overview
So instead of applying the timeout to everything, and then opting
certain long-running calls out of it, this just opts-in the
"GetChanges" call to the timeout. This should cover the failure mode
that was originally solved for chrome's bots. And also bump the
timeout to 30s since we don't trust all get-changes queries to
reliably finish in under 10s.
This also bumps the step level timeout for the query in tryserver
recipe_mod to 8 min since there could be ~5min worth of sleeping +
150s worth of waiting+timing-out. So worst case the step will now
take 8min to fully exhaust all timeouts/sleeps.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: b/278083716
Change-Id: Ib366e004e0bb07297ba732590d488cae779e38ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4426524
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
When gerrit's availability drops, chrome's builds see an excessive
amount of failures, eg:
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359780/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359594/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359564/overviewhttps://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359630/overview
Seemingly all failures occur in either the `gerrit fetch current CL
info` step or the `gerrit changes` step. Both steps have a 60s timeout
and shell out to depot_tools' gerrit_client.py. That script essentially
makes a single http call to gerrit. That request has no configured
timeout. So when gerrit's MIA and the call hangs indefinitely, so too
will the step hang. 60s after that, the step timeout is reached, and the
entire build crashes with an infra-failure.
However, one single retry has been shown to sufficiently work around
at least one instance of that failure:
dea9a6eca26f690ce10a959701a8fb51ad/+/build.proto
So this incorporates timeouts into the requests made by
gerrit_util.py. Each request is given a 10s timeout, which should be
enough for most/all gerrit calls. (Both steps have a p90 of less than
1sec.) When a timeout is reached, the script will automatically retry
up to 4 times.
This also bumps the timeouts of the step calls to gerrit_client.py to
6min since the script can now take up to 5 minutes to fail, ie:
the sequence of sleeps is roughly 10s, 20s, 40s, 80s, 160s, which is
about 5min. So a 6min timeout should cover all those retries.
This also passes in "--verbose" to all step calls to this script, so
the logging that prints out the retry + sleep log lines will be
visible in build logs.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: 1432638
Change-Id: I9dc47f4beeda3783ae4f9152bd29ee441ac3e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4420526
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
This reverts commit 19794fe91f.
Reason for revert: Doesn't fix the cache.
Original change's description:
> Reset origin/HEAD when it points to master
>
> This change tries resetting origin/HEAD when it points to master and the ref cannot be found. Post master->main migration, some cached repositories seem to point origin/HEAD to master.
>
> Bug: 1418866
> Change-Id: I83942901b105cace4d51deeac7e400f98d200168
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4296441
> Reviewed-by: Fumitoshi Ukai <ukai@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: 1418866
Change-Id: I4c912f8401afb6fb0c2149c277778bdee9fa5d76
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4304240
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This change tries resetting origin/HEAD when it points to master and the ref cannot be found. Post master->main migration, some cached repositories seem to point origin/HEAD to master.
Bug: 1418866
Change-Id: I83942901b105cace4d51deeac7e400f98d200168
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4296441
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
The python module is deprecated in favor of using the step module with
an explicit python binary. Switching the code to use the step module and
remove DEPS on recipe_engine/python removes warnings when running recipe
tests in the repo.
Change-Id: If3c9d5d6aff5aab69898b535399accbef92f5c0b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3880624
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts https://crrev.com/c/3697958 and https://crrev.com/c/3704921.
One reason is that recipe bundle is not designed to be mutated.
Another reason is the presence of gomacc binary under .cipd_bin causes an unexpected side effect. https://crbug.com/1337048
Bug: 931218, 1337048
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: Iaea12a0ad9246c0781035aa499cc6c0bb42d71c3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3715873
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8815259687168433617
Please review the expectation changes, and LGTM+CQ.
recipe_engine:
4cb1bf0fb0
4cb1bf0 (gbeaty@chromium.org)
Don't set the -u flag when running python unbuffered.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I0859db0f9db1c7548068073300150e979ac460b4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3621157
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Auto-Submit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Reviewed-by: Garrett Beaty <gbeaty@google.com>
This is a reland of commit 053817260a
The change was not related to the errors that it was reverted for, so no
changes are necessary.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: I80efeee8a2b951df43b00d89bb596dc2cf2fcec7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3582182
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
This reverts commit 053817260a.
Reason for revert: we reverted got_revision change, and this change
itself may be causing some other failues.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Change-Id: Ie975dfe39e7b8997336761e12f53f3046359d825
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3578820
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
The bot_update.py script sets a got_revision property by default even if
it is not present in the reverse revision mapping. This results in an
uncaught exception when set_output_commit is set to True if got_revision
isn't present in the reverse revision mapping, but it isn't caught until
production because the test API doesn't match that behavior. This change
updates the test API method to match the behavior of the script.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
The step_result json can currently be empty if the orchestrator build
is canceled while running bot_update. Trying to do
step_result.json.output in the finally block raises an "Uncaught
exception AttributeError".
Ex: https://ci.chromium.org/ui/p/chromium/builders/try/win10_chromium_x64_rel_ng/1121568/overview
This change checks for the json attr before trying to read json.output.
Bug: 1305332
Change-Id: I1dc277642ece1b96144be6381a7c4b8426cb2bc6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3518372
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This reverts commit 8b64ddec2a.
Reason for revert: it appears we need this, since .DEPS.git
specifies custom dependencies needed for tryjob.
Original change's description:
> Remove unused functionality in bot_update.py
>
> .DEPS.git has been removed in 2014 and no longer supported. It was
> deleted in https://codereview.chromium.org/497313003 (August 2014). Last
> entry in chromium/src is from tag 44.0.2403.47 in June 2015.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chrome_release
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: I3ffb8813af867df959098c02d399c8d745894aae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3439573
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Change-Id: I36bcd352a984b4b762096197a384e86347418290
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3448707
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
.DEPS.git has been removed in 2014 and no longer supported. It was
deleted in https://codereview.chromium.org/497313003 (August 2014). Last
entry in chromium/src is from tag 44.0.2403.47 in June 2015.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: I3ffb8813af867df959098c02d399c8d745894aae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3439573
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
gclient removed syntax validation flag in 2020:
https://crrev.com/c/2083589, so this can safely be removed.
R=aravindvasudev@google.com, gavinmak@google.com
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build
Change-Id: I2aced6cdb01d571ca361cc57e7c8715c9ca40552
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3419278
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Those paremeters are deprected for over 4 years now. I found no usages
while searching for output_manifest in source.chromium.org.
R=machenbach@google.com
Bug: 1289274
Change-Id: I7d8f8ee29fdb046b255c96ad26f8dc7129b1f83a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3402741
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
The blamelist pins are produced by iterating over a dictionary which
yields a different order between python2 and python3. The blamelist pins
will now be produced with a consistent ordering between python versions
by sorting on the revision name.
Change-Id: Ieb6c4dfe0aacea83c4a86a3a10380cc2f3839114
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3352250
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Only hanging process recorded so far is gsutil. Sending sigterm doesn't
result in successfully terminating the process. Instead, use sigkill.
R=gavinmak@google.com
Bug: 1255228
Change-Id: Iad354284422676df804c68cb6f18c7db723bf862
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3232250
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
It's generaly not useful and makes it harder to understand bot_update
logs. If we need to debug specific builders, we can set environment
variables in the recipe itself.
R=apolito@google.com
Bug: 1255228
Change-Id: I18fc6ee8cb15b30bc3a4422ddd04979189bacf64
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3221250
Commit-Queue: Anthony Polito <apolito@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
Patch failures shouldn't be treated as infra_failures (as long as patch
was successfully downloaded).
Prior to https://crbug.com/1207685, bot_update patch failurs were set as
successful step to work around buildbucket plugin requirements.
One alternative approach is to add necessary information into patch
failure step, by editing its presentation.
R=gavinmak@google.com
Bug: 1207685, 1255228
Change-Id: I696356d40a80536c3efbca99091536b8b25deb40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219890
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Gavin Mak
recipe-roller
Yiwei Zhang
Garrett Beaty
Aravind Vasudevan
Joanna Wang
Ben Pastene
Joanna Wang
Takuto Ikuta
Junji Watanabe
Ravi Mistry
Josip Sokcevic
Jack Neus
Stephanie Kim
Brian Ryner
Anthony Polito