This reverts commit 72a245e4c6.
Reason for revert: crbug.com/1022617
Original change's description:
> Fetch the input commit ref when both ref and id are specified.
>
> This causes --refs refs/head/master to be added to bot_update calls when
> the input build message contains a commit id and the refs/head/master
> ref. This is an effective no-op.
>
> When bot_update clones a git repository, it results in HEAD referring to
> master in contrast to the non-clone case which results in a detached
> HEAD. This prevents fetching refs/heads/master, so detach the HEAD after
> cloning.
>
> Bug: 922150
> Change-Id: I357a78305020218e9f5d350832a706b8f2b27967
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: infra
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1900735
> Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
TBR=ehmaldonado@chromium.org,gbeaty@chromium.org
Change-Id: Ibd6c6ed272b0c98f82be50cefa521aaee0444f11
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 922150, 1022617
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1904099
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
This causes --refs refs/head/master to be added to bot_update calls when
the input build message contains a commit id and the refs/head/master
ref. This is an effective no-op.
When bot_update clones a git repository, it results in HEAD referring to
master in contrast to the non-clone case which results in a detached
HEAD. This prevents fetching refs/heads/master, so detach the HEAD after
cloning.
Bug: 922150
Change-Id: I357a78305020218e9f5d350832a706b8f2b27967
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1900735
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
There are 3 layers modified, starting from the bottom up:
1. git_cache.py populate
Now takes a --no-fetch-tags option. If specified, the cache will not
fetch updated tags from the server by passing --no-tags to git fetch.
This prevents the server from sending all the tag refs. In chromium
this prevents significant time bottlenecks dealing with 10k+ tags.
2. bot_update.py
bot_update has to deal with multiple git repos, it has the root repo
that is checked out through git-cache, as well as repos synched via
DEPS and gclient.
The script now takes a --no_fetch_tags option. If specified,
the git-cache populate call will include --no-fetch-tags. Otherwise, it
won't. So this controls (for chromium) if fetches to the src.git server
are passed --no-tags.
3. bot_update/api.py
This is the entry point for recipes and also has to deal with multiple
git repos. The behaviour at this point is not changed from the default.
A |no_fetch_tags| parameter is added to ensure_checkout() that defaults
to False.
This CL is a refactor with no intended behavior change.
The next step will be to change the chromium build repo to pass True
for ensure_checkout(no_fetch_tags) on chromium trybots.
This represents solution #2 in
https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit#
Bug: 1019824
Change-Id: I935430603299daa9e301a95a5184c0ce486fd912
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
This reverts commit 512f92b731.
Reason for revert: bot_update is having infra failure on ci bots.
For example https://ci.chromium.org/p/chromium/builders/ci/Closure%20Compilation%20Linux/249684.
Original change's description:
> Fetch the input commit ref when both ref and id are specified.
>
> This causes --refs refs/head/master to be added to bot_update
> calls when the input build message contains a commit id and the
> refs/head/master ref. This is an effective no-op.
>
> Bug: 922150
> Change-Id: I119d9b3cede727713a0b66bff83b5ee4e1848790
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1884457
> Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
> Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
TBR=tandrii@google.com,gbeaty@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 922150
Change-Id: Ie9275c21827688f59449f052c768452a8652e139
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891000
Commit-Queue: Haiyang Pan <hypan@google.com>
Reviewed-by: Haiyang Pan <hypan@google.com>
This causes --refs refs/head/master to be added to bot_update
calls when the input build message contains a commit id and the
refs/head/master ref. This is an effective no-op.
Bug: 922150
Change-Id: I119d9b3cede727713a0b66bff83b5ee4e1848790
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1884457
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
{_resolve_fixed_revisions} resolves every revision property. After a test
failure, we don't want to resolve every revision always.
This CL allows updating a subset of revisions for a new checkout.
R=machenbach@chromium.orgTBR=tandrii@chromium.org
Bug: 950731, 982711
Change-Id: I3cdc5cdb882009886b82557427fdf9b02e860b39
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1720929
Commit-Queue: Tamer Tas <tmrts@chromium.org>
Auto-Submit: Tamer Tas <tmrts@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
got_revision and got_revision_cp point to the commit that
Build.output.gitiles_commit should point to.
Therefore, derive output commit from these properties.
Derive output ref from got_revision_cp if it available.
Otherwise, try to read it from input commit
Bug: 940214
Change-Id: I108d782ad93c70114581830d7427b95d3efd9d98
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1588506
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
This reverts commit 9198ef8ede.
Reason for revert: Spec revert: appears to be causing large number of chromium CQ failures on patch application.
Original change's description:
> [bot_update] Default to refs/heads/master
>
> When revision is not specified, currently HEAD is checked out.
> Change the default to refs/heads/master
>
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Bug: 940214
> Change-Id: I8be59095d90447ea898d5247a6433f0734f5e63b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1554862
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
TBR=nodir@chromium.org,tandrii@chromium.org,jbudorick@chromium.org
Change-Id: Icafd817d8f044a4347342bd85dc97d193329a704
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 940214
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1555514
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
When revision is not specified, currently HEAD is checked out.
Change the default to refs/heads/master
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 940214
Change-Id: I8be59095d90447ea898d5247a6433f0734f5e63b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1554862
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Add set_output_commit param to bot_update.ensure_checkout, false by default.
If true, call api.buildbucket.set_output_gitiles_commit to set the output commit
of the build. This generally supersedes got_revision and got_revision_cp.
Bug: 940214
Change-Id: I5ffc790a6b12c8bf4a56469b3ecc567141c0d735
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1548256
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
This has been low-key driving me bananas for a while, and I've seen
it confuse sheriffs etc many times.
Change-Id: If46ab55821447a3a70a3f0ea774c017f29fc7400
Reviewed-on: https://chromium-review.googlesource.com/c/1286819
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
It has been True by default for a while, and there is no need to override it.
Bug: 891917
Change-Id: I2598a2230b0ea38a647a533757331c541b871971
Reviewed-on: https://chromium-review.googlesource.com/c/1260057
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Use get_gerrit_patch_root in bot_update.ensure_checkout.
This fixes some tests expectations that expected CI builds to pass --patch_root
flag.
Also unify "root" and "patch_root" variables in ensure_checkout. Simpler.
This CL requires nontrivial rolls in all downstream packages because all
packages have incorrect expectations and/or tests.
In particular, this removes --patch_root flag from non-tryjob test
cases. Some tests are supposed to test tryjob scenario, but they use
properties.generic instead of properties.tryserver (or buildbucket.try_build)
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skiabuildbot
Bug: 694348
Change-Id: Ibc594fa42e3b4d2823672b41dda2d97154b10d38
Reviewed-on: https://chromium-review.googlesource.com/1231280
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Update set_patch_project_revision to use repo_path_map instead of
patch_projects.
Also rename to set_patch_repo_revision.
Bug: 694348
Change-Id: Iba31f4910f8282160000b8fcef197052597530a4
Reviewed-on: https://chromium-review.googlesource.com/1226107
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Add a shortcut for getting local path to the patch repo.
In bot_update.api._destination_branch, start using it instead of reading
gclient config directly. This removes one more patch_project usage.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Bug: 694348
Change-Id: If05371d0f6d11be6803ffa8392e029f5860dda25
Reviewed-on: https://chromium-review.googlesource.com/1217670
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Add tryserver.api.gerrit_change_info that returns detailed info about the
current Gerrit CL. It makes a Gerrit RPC in the background on the first read.
This step will be present in all tryjobs.
This mechanism can be used for extra details about the current CL.
This CL starts to use this in bot_update for gerrit_ref, and stops using
patch_ref property, and also update bot_update to use when the destination
branch is needed.
Also this CL removes no longer used properties in bot_update,
namely most of patch_* properties.
build.git manual roll: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1217626
Recipe-Manual-Change: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: release_scripts
R=tandrii@chromium.org
Bug: 694348
Change-Id: I945d061b74278c3d1d227578abe16a64cdaca647
Reviewed-on: https://chromium-review.googlesource.com/1213557
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
CrOS chromium-pfq builder uses bot_update, but checks out Chromium on its own,
effectively ignoring input commit in bot_update.
To resolve a P0, add ignore_input_commit parameter.
Long term, bot_update API must be redesigned.
TBR=tandrii@chromium.org
Bug: 882094
Change-Id: I4f43348e3d5f81bf603dd2c110829bce58acad16
Reviewed-on: https://chromium-review.googlesource.com/1216787
Reviewed-by: Lann Martin <lannm@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Use self.m.buildbucket.build.input.gitiles_commit.id instead.
It encapsulates revision and parent_got_revision properties.
Also, honor host and project of the specified commit.
This enables overriding non-first gclient solution at the build-level
for arbitrary buildbucket builds.
Bug: 877161
Change-Id: I3f92f04ca84a1d30cbbb2670f62bea2bd9ee5932
Reviewed-on: https://chromium-review.googlesource.com/1200066
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
When checking if buildbucket.build_input.gitiles_commit exists, use
HasField protobuf method, as opposed to truth check. A protobuf message
field is always true.
Bug: 877161
Change-Id: Ie4a7e22b985776441a74283b09646e8837c326f1
Reviewed-on: https://chromium-review.googlesource.com/1196930
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Today, whenever LUCI scheduler sets gitiles_commit, a revision property
is also always set, thus this CL will have no effect on today's production.
The goal of CL is to pave the way to avoid this duplication. For starters,
one could start real writing tests with
+ api.buildbucket.ci_build(..., revision='sha1')
without the need to duplicate the same revision as a property. All
recipes which WAI with such input are thus ready to work w/o 'revision'
property. After that, we'll be able to stop the duplication in LUCI scheduler.
R=hinoka
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Change-Id: I1773b1c3d2a98a9dbd1cfdd9ac2b4fb99745ee99
Reviewed-on: https://chromium-review.googlesource.com/1123713
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
shallow mode is a special git mode that only checks out a shallow copy
of git so the full history is not downloaded. It is not well supported
by googlesource.com servers because git has to do a giant negotiation
with the server and transmit every single commit that the local checkout
has or doesn't have (instead of just the latest commit of each ref),
which ends up taking more than 10x longer than a non-shallow checkout.
R=ehmaldonado, hinoka, iannucci
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 855137
Change-Id: I5f9e31f8b2730cf80b79bfd0e08201c33eb942f5
Reviewed-on: https://chromium-review.googlesource.com/1110450
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This is a reland of 3806b7fbd0
Original change's description:
> bot_update: Fetch references to be synced.
>
> If a solution is to be synced to a reference, such as
> refs/branch-heads/#### fetch it so that it can later be
> checked out.
>
> I tried this change on a LUCI task that runs chromium_presubmit
> for crrev.com/c/1048868. The bot_update works as expected:
> https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com
>
>
> Bug: 840445
>
> Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/1049007
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Bug: 840445
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I64a97aafe5fab9a26c478262e9ca104dc59e61f5
Reviewed-on: https://chromium-review.googlesource.com/1052387
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
This reverts commit 3806b7fbd0.
Reason for revert: Broke downstream tot bots
Original change's description:
> bot_update: Fetch references to be synced.
>
> If a solution is to be synced to a reference, such as
> refs/branch-heads/#### fetch it so that it can later be
> checked out.
>
> I tried this change on a LUCI task that runs chromium_presubmit
> for crrev.com/c/1048868. The bot_update works as expected:
> https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com
>
>
> Bug: 840445
>
> Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/1049007
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
TBR=mmoss@chromium.org,machenbach@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org
Change-Id: If94df8684e634504d8a0c11b2aa5862a458ba19f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 840445, 841306
Reviewed-on: https://chromium-review.googlesource.com/1052187
Reviewed-by: agrieve <agrieve@chromium.org>
Commit-Queue: agrieve <agrieve@chromium.org>
If a solution is to be synced to a reference, such as
refs/branch-heads/#### fetch it so that it can later be
checked out.
I tried this change on a LUCI task that runs chromium_presubmit
for crrev.com/c/1048868. The bot_update works as expected:
https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com
Bug: 840445
Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/1049007
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This is needed for release builders to checkout the release tags that
will be created when the buildspec branch/release info is migrated out
to the regular repos.
BUG=825063
R=jbudorick@google.com
Change-Id: I5f0e4cdd94de11c59f9b4a5c11aabea129e81523
Reviewed-on: https://chromium-review.googlesource.com/1036526
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
This reverts commit 355b11a31f.
Reason for revert:
Easier to do this another way that doesn't cause expectation changes,
so it doesn't require non-trivial rolls.
Original change's description:
> bot_update: Patch on gclient by default.
>
> See [4] for the design doc for this change.
>
> We're moving the patching logic from bot_update to gclient.
> This way we can test patches of dependencies that modify their DEPS
> files as well, for example when ANGLE uploads a patch that modifies
> DEPS files to be tested by a Chromium trybot.
>
> Chromium Linux CQ and Angle trybots are already running this way
> (see [1] and [2]) and a PSA has already been sent [3]
>
> [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
>
> Bug: 643346
> Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: release_scripts
> Reviewed-on: https://chromium-review.googlesource.com/1014356
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
TBR=agable@chromium.org,ehmaldonado@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 643346
Change-Id: I0e694b0e398770b8bd2e5a3170e158a7e54c5a61
Reviewed-on: https://chromium-review.googlesource.com/1029310
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This adds a --enable-gclient-experiment flags that tells bot_update.py
to skip applying the patch, and instead forward the flags to gclient.
Bug: 643346
Change-Id: Ia4275a126e6adba54dfcc894d224c50c166db90e
Reviewed-on: https://chromium-review.googlesource.com/962938
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This method hasn't been supported for a long time now (and besides, its
only for Rietveld).
R=agable@chromium.org, tandrii@chromium.org
Recipe-Manual-Change: infra
Change-Id: Ie6e63834dca67962db29f2cb407950ed85db55a7
Reviewed-on: https://chromium-review.googlesource.com/957832
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
This partially reverts commit d51ed57edb.
Reason for revert:
New git client for windows was rolled including fix for slow `git fetch`.
I guess smaller pack limit causes frequent bootstrap taking 2~3 minutes longer than
the case it does not happen.
Let me see what happen if we increase pack limit 9 -> 30.
I will increase this to 50 if this won't cause regression again.
Original change's description:
> git_cache: lower max num of .pack files before re-bootstrap on Win.
>
> It used to be 50, I think ~9 gives best results for Chromium on Win:
> on golo VM, it takes <4 minutes to re-boostrap + git fetch small
> delta, assuming zipped git checkout for bootstrap is fresh (~1day).
>
> For other repos, which are significantly smaller, this change should
> have minor effect if at all.
>
> Test: I tested this using `led` tool on Win7 machines running LUCI
> stack extensively. For example,
>
> * https://ci.chromium.org/swarming/task/3a0102e8c8657410
> shows case with few .pack files, hence just 1 fetch
>
> * https://ci.chromium.org/swarming/task/3a010282f9fd8010
> shows case with 39 .pack files and so bootstrapping + fetch.
> If you look at prior tasks on the same VM, you'd find this:
> https://ci.chromium.org/swarming/task/39ffe843d01ed010
> which spent 8 minutes doing 1 incremental fetch with 39 .pack
> files.
>
> **Troopers/Sheriffs**: This change is safe to revert.
> However, beware that you should also at the same time revert the recipe
> roll of this CL to the repo, in which the failed builder's recipe is
> located, typically `chromium/tools/build`.
>
> Bug: 749709
> Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
> Reviewed-on: https://chromium-review.googlesource.com/787174
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 749709
Change-Id: I3052abe4a9b53277a60c0791a85355e7a0bbdf8f
Reviewed-on: https://chromium-review.googlesource.com/823544
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This will be used by various things in build and build_internal which
"need" a depot_tools on PATH (though I suspect most of those should be
refactored to use the depot_tools in chromium/src.git).
R=tandrii@chromium.org
Bug: 789808
Change-Id: I5d0590ccdc5482bd8e3048ad46a39b5aedacbddb
Reviewed-on: https://chromium-review.googlesource.com/828465
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
It used to be 50, I think ~9 gives best results for Chromium on Win:
on golo VM, it takes <4 minutes to re-boostrap + git fetch small
delta, assuming zipped git checkout for bootstrap is fresh (~1day).
For other repos, which are significantly smaller, this change should
have minor effect if at all.
Test: I tested this using `led` tool on Win7 machines running LUCI
stack extensively. For example,
* https://ci.chromium.org/swarming/task/3a0102e8c8657410
shows case with few .pack files, hence just 1 fetch
* https://ci.chromium.org/swarming/task/3a010282f9fd8010
shows case with 39 .pack files and so bootstrapping + fetch.
If you look at prior tasks on the same VM, you'd find this:
https://ci.chromium.org/swarming/task/39ffe843d01ed010
which spent 8 minutes doing 1 incremental fetch with 39 .pack
files.
**Troopers/Sheriffs**: This change is safe to revert.
However, beware that you should also at the same time revert the recipe
roll of this CL to the repo, in which the failed builder's recipe is
located, typically `chromium/tools/build`.
Bug: 749709
Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
Reviewed-on: https://chromium-review.googlesource.com/787174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This reverts commit 47b67c426b.
Reason for revert: Bug in bot_update.py resource, breaks recipe roller.
Original change's description:
> Reland "bot_update recipe: Upload source manifest"
>
> This reverts commit c3d1208d5c.
>
> Also:
> * Instead of replacing "manifest", just add a new "source_manifest"
> to the output JSON. This allow transition without breakage.
> * Change the test api so test for recipe output changes.
>
> The plan is to land this first, switch all downstream to "source_manifest",
> and then remove the original "manifest" key.
>
> Bug: 772529,776299
> Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: release_scripts
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Reviewed-on: https://chromium-review.googlesource.com/731378
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>
TBR=iannucci@chromium.org,hinoka@chromium.org
Change-Id: I7a4ee904075e8b75b8a47f9ef0cd8a633af85a9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529, 776299
Reviewed-on: https://chromium-review.googlesource.com/748312
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This reverts commit c3d1208d5c.
Also:
* Instead of replacing "manifest", just add a new "source_manifest"
to the output JSON. This allow transition without breakage.
* Change the test api so test for recipe output changes.
The plan is to land this first, switch all downstream to "source_manifest",
and then remove the original "manifest" key.
Bug: 772529,776299
Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/731378
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This is a step away from patch_project, which was a holdover
from Rietveld. Instead of mapping "projects" (which are
poorly-defined and not guaranteed to be unique) to subpaths,
instead map repository urls (which are at least unique).
Eventually we may be able to compute this directly from DEPS
instead of hardcoding the mapping here, but at least this is
a step in the right direction.
Bug: 765633
Change-Id: Idd65984fc6edefcbedb0438d38c2338b10b7e8e5
Reviewed-on: https://chromium-review.googlesource.com/690776
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
The only user of this method needs to apply two patches from two
different repos in a try job. Extending the method to handle this.
BUG=628259
R=agable@chromium.org
Change-Id: Ia119780d1323add0094631e5c291b0e7a573e6e5
Reviewed-on: https://chromium-review.googlesource.com/588008
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This was removed in downstream recipes after it was turned on by default.
Bug: 740456
Change-Id: Idae1905af97333970c8b31508634f8b71550c6f9
Reviewed-on: https://chromium-review.googlesource.com/574030
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
The previously passed branch:HEAD notation is understood by bot_update
for the main project, but not by gclient for deps'ed projects.
We don't need the colon notation at all as passing a branch implies
using the HEAD of that branch.
Bug: 740456
Change-Id: I95eb88f0de2e06bee8a3e7db24c4ad85cdb76d3e
Reviewed-on: https://chromium-review.googlesource.com/566866
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
This makes bot_update query Gerrit when syncing for Gerrit tryjobs.
The query will establish the actual destination branch of the CL,
which can be different from master (e.g. a feature branch).
Bot_update will ensure to use this destination branch for the repo
that corresponds to the CL's project. Both the main project or a
deps'ed project work.
Initially, this lives behind a flag that can be controlled in
downstream recipes. Eventually we'll set this to default after a
gradual roll-out.
Branches in branch-heads are not supported yet.
Bug: 740456
Change-Id: I4a0d50e2ca8fe90f8d29964a3ffab17291f7be60
Reviewed-on: https://chromium-review.googlesource.com/566824
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
"bot_update" currently passes a full "gclient" specification as a
command-line argument to "bot_update.py". This file contains all sorts
of junk that clutters the command-line, and revealed a batch file
escaping bug that seems to be present in the recipe engine. Change this
so that "bot_update" passes the spec as file content instead of a
command-line flag.
Also, "bot_update" apparently called "--spec", which optparse kindly
recognized as the real flag, "--specs". This has always(?) been a bug,
and is no longer a problem.
BUG=chromium:738348
TEST=expectations
Change-Id: I5324b94dd5ef78503e7cfdc28b5fde8f434e4359
Reviewed-on: https://chromium-review.googlesource.com/558207
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Prefix "depot_tools" to PATH instead of appending. This will ensure that
"depot_tools" pulls resources from the recipe checkout as a preference,
not as a last resort.
Use "env_prefixes" instead of "env" to prefix PATH.
Remove Git speed limit settings, which are taken care of by the Git
wrapper now.
BUG=chromium:738176
TEST=expectations
R=hinoka@chromium.org, iannucci@chromium.org
Change-Id: I2e509a331266e867be323bc70e3bc0f08f85246f
Reviewed-on: https://chromium-review.googlesource.com/556287
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Currently, "bot_update" relies on a BuildBot cleanup mechanism and, to a
lesser extent, the standard BuildBot directory layout. Both of these are
problematic when projecting it into other circumstances, notably
"remote_run" and LUCI.
Have "bot_update" handle its own cleanup. It will now choose a cleanup
directory within the hierarchy of its checkout, and explicitly purge it
prior to execution if it exists. This enforces its expected behavior in
all circumstances and removes its expectations of the greater checkout
layout.
Export "cleanup_dir" via "infra_paths" to point to "build.dead" when
running on BuildBot builds. Otherwise, it is a default directory which,
on Kitchen, is ephemeral by design.
BUG=chromium:725631
TEST=expectations
Change-Id: I664434c542a25aaa7ff3eac216208a2425730fde
Reviewed-on: https://chromium-review.googlesource.com/528057
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Some recipes interacting with older revisions (e.g. bisect)
will need this.
Bug: 570091
Change-Id: I38e5ffa2db1a9bfae3667f015f00977c32ebe51e
Reviewed-on: https://chromium-review.googlesource.com/519407
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit c91686c087.
Reason for revert: Reland a light version of the original CL that keeps a dummy flag around.
Original change's description:
> Revert "Remove obsolete bot_update flag"
>
> This reverts commit 72048266d5.
>
> Reason for revert: Skia depends on it. Need to clean that up first. Currently blocks roll:
> https://skia-review.googlesource.com/c/14145/
>
> Original change's description:
> > Remove obsolete bot_update flag
> >
> > Using the manifest is the default since a long time. The flag was only introduced to smoothly switch it on. Now the flag is not needed anymore.
> >
> > Bug: 398105
> > Change-Id: I4eb1c98e26e025a69f8b0a7c48877d78c031702e
> > Reviewed-on: https://chromium-review.googlesource.com/485659
> > Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> > Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
> > Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> >
>
> TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org,sergiyb@chromium.org,chromium-reviews@chromium.org
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Change-Id: Iffc4d8076abd3c92feebfd7f3ef88be7ce34b692
> Reviewed-on: https://chromium-review.googlesource.com/486639
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
>
R=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org,sergiyb@chromium.org,chromium-reviews@chromium.org
Bug: 398105
Change-Id: I817d446f8a8eff84c5d7143ac2b9c527a40991f1
Reviewed-on: https://chromium-review.googlesource.com/486640
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
This reverts commit 72048266d5.
Reason for revert: Skia depends on it. Need to clean that up first. Currently blocks roll:
https://skia-review.googlesource.com/c/14145/
Original change's description:
> Remove obsolete bot_update flag
>
> Using the manifest is the default since a long time. The flag was only introduced to smoothly switch it on. Now the flag is not needed anymore.
>
> Bug: 398105
> Change-Id: I4eb1c98e26e025a69f8b0a7c48877d78c031702e
> Reviewed-on: https://chromium-review.googlesource.com/485659
> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
> Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
>
TBR=phajdan.jr@chromium.org,machenbach@chromium.org,tandrii@chromium.org,sergiyb@chromium.org,chromium-reviews@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Change-Id: Iffc4d8076abd3c92feebfd7f3ef88be7ce34b692
Reviewed-on: https://chromium-review.googlesource.com/486639
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Using the manifest is the default since a long time. The flag was only introduced to smoothly switch it on. Now the flag is not needed anymore.
Bug: 398105
Change-Id: I4eb1c98e26e025a69f8b0a7c48877d78c031702e
Reviewed-on: https://chromium-review.googlesource.com/485659
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
A few rare recipes use two or more solutions side-by-side.
This allows switching one solution in isolation to use the new mapping.
Bug: 713356
Change-Id: Id51f2f777681401843004577a4de65a6c5d1e4a6
Reviewed-on: https://chromium-review.googlesource.com/484341
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This is the first step in reversing got_revision_mapping.
This CL:
- Adds a new configuration got_revision_reverse_mapping.
- Ensures only the old or the new configuration is used.
- Transforms the old to the new configuration within bot_update.
In follow up CLs we will migrate all clients to the new mapping, then delete the old and rename the new again.
Bug: 713356
Change-Id: Icdbd6a3748c6e5399b9c4867a3d8292973325160
Reviewed-on: https://chromium-review.googlesource.com/483479
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit e8acd40a11.
Reason for revert: broke windows builds :(
Original change's description:
> bot_update: pin depot_tools
>
> gclient runhooks often calls scripts that depend on scripts in depot_tools.
> Currently bot_update only ensures that depot_tools is present in $PATH, but
> otherwise uses whatever depot_tools already specified in $PATH. We don't pin
> depot_tools before running a recipe, so using whatever depot_tools in $PATH
> is not hermetic.
>
> Insert the path to the pinned depot_tools to the beginning of $PATH
>
> R=iannucci@chromium.org
>
> Bug:
>
> Change-Id: I31d04ba2ec7c409f89d58a51502ce3565e3a02a1
> Reviewed-on: https://chromium-review.googlesource.com/481072
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
>
TBR=iannucci@chromium.org,nodir@chromium.org,phajdan.jr@chromium.org,chromium-reviews@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Change-Id: I1a70f0f54e9b9cea7fc2f9897e62f558204a4684
Reviewed-on: https://chromium-review.googlesource.com/481440
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
gclient runhooks often calls scripts that depend on scripts in depot_tools.
Currently bot_update only ensures that depot_tools is present in $PATH, but
otherwise uses whatever depot_tools already specified in $PATH. We don't pin
depot_tools before running a recipe, so using whatever depot_tools in $PATH
is not hermetic.
Insert the path to the pinned depot_tools to the beginning of $PATH
R=iannucci@chromium.org
Bug:
Change-Id: I31d04ba2ec7c409f89d58a51502ce3565e3a02a1
Reviewed-on: https://chromium-review.googlesource.com/481072
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Stephen Martinis
Garrett Beaty
danakj
Haiyang Pan
Marc-Antoine Ruel
Andrii Shyshkalov
Tamer Tas
Edward Lemur
Oleh Prypin
Nodir Turakulov
John Budorick
Andrii Shyshkalov
William Hesse
Nodir Turakulov
Takuto Ikuta
agrieve
Michael Moss
Ben Pastene
Robert Iannucci
Takuto Ikuta
Ryan Tseng
Ryan Tseng
Michael Achenbach
Aaron Gable
Sergey Berezin
Dan Jacques
Dan Jacques
Paweł Hajdan, Jr