This method hasn't been supported for a long time now (and besides, its
only for Rietveld).
R=agable@chromium.org, tandrii@chromium.org
Recipe-Manual-Change: infra
Change-Id: Ie6e63834dca67962db29f2cb407950ed85db55a7
Reviewed-on: https://chromium-review.googlesource.com/957832
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Also, some lines went over 80chars, so I ran 'yapf --style chromium' which
formatted some unrelated things.
BUG=905285
R=dpranke
Change-Id: Iee5f46d88a6e9782612cc4f9e5a2cb72d62ab6af
Reviewed-on: https://chromium-review.googlesource.com/907736
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
This reverts commit fa573785df.
Reason for revert: broke CQ on branches.
Example: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/28220
Original change's description:
> bot_update: allow rebasing the patch onto an older revision.
>
> When applying a rebase, we normally go from an older base commit to a newer one.
>
> A ---- B ---- C ---- D -origin/master
> \
> E -branch
>
> In this case, `git rebase D` would certainly work as expected. However,
> writing `git rebase B` would NOT get us to the following state:
>
> A ---- B ---- C ---- D -origin/master
> \
> E' -branch
>
> In fact, it would have no effect.
>
> This article http://matthew-brett.github.io/pydagogue/rebase_without_tears.html
> explains the general invocation as
> > `git rebase --onto <graft-point> <exclude-from> <include-from>`
> > If you don’t specify --onto, <graft-point> defaults to <exclude-from>
>
> So what's happening is, by writing `git rebase B` we're rebasing onto B,
> excluding commits that are in B. Commit C is not "in" B so it is kept and we're
> back to the starting point.
>
> So I suggest to change the invocation to `git rebase --onto B origin/master`,
> which rebases onto B, excluding commits that are in origin/master. This works
> more generally and allows rebasing "backwards".
>
> Bug: None
> Change-Id: I68e4d805811530b585550bc75099354fef4e9c15
> Reviewed-on: https://chromium-review.googlesource.com/904004
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Oleh Prypin <oprypin@chromium.org>
TBR=iannucci@chromium.org,tandrii@chromium.org,oprypin@chromium.org
Change-Id: I7243641d84428bf4504ee798bf31ad0afbfd4865
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: None
Reviewed-on: https://chromium-review.googlesource.com/905862
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Goal: reduce number of git fetches from remote to the minimum possible.
Minimum possible is either:
* 0 if desired revision is pinned (!= HEAD) and already exists in cache
* 1 if desired revision is HEAD
* >= 2 if revision is pinned but we talk temporarily
out-of-date git server not yet having it.
This CL achieves the above.
No fetch:
https://ci.chromium.org/swarming/task/39dad20230c84110?server=chromium-swarm.appspot.com
bot_update takes <1min, but it used to take at least 2. The steps are now like this:
...
===Running git cat-file -e 33bf8a94dcd5e0abbdf83e4afaed24b5180e3eb2===
In directory: /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
===Succeeded in 0.0 mins===
===Running /b/swarming/w/ir/cipd_bin_packages/bin/python -u /b/swarming/w/ir/recipe-checkout-dir/depot_tools/git_cache.py exists --quiet --cache-dir /b/swarming/w/ir/cache/git https://chromium.googlesource.com/chromium/src.git===
In directory: /b/swarming/w/ir/cache/builder/linux
/b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
===Succeeded in 0.0 mins===
===Running git remote set-url origin /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src===
In directory: /b/swarming/w/ir/cache/builder/linux/src
===Succeeded in 0.0 mins===
===Running git fetch origin===
In directory: /b/swarming/w/ir/cache/builder/linux/src
From /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
003eedbfee47..5210107a20e8 master -> origin/master
===Succeeded in 0.0 mins===
===Running git checkout --force 33bf8a94dcd5e0abbdf83e4afaed24b5180e3eb2===
...
Fetch & success:
https://ci.chromium.org/swarming/task/39db022b874ffd10?server=chromium-swarm.appspot.com
Fetch & retry because missing rev:
https://ci.chromium.org/swarming/task/39db04ce8b4b8910?server=chromium-swarm.appspot.com
Fetch because rev=HEAD:
https://ci.chromium.org/swarming/task/39db0c0dd5ab5d10?server=chromium-swarm.appspot.com
This has already been adopted in gclient.
Bug:
Change-Id: Id99892b62719fdf3f7e6e59058986d1500384f8d
Reviewed-on: https://chromium-review.googlesource.com/771591
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
When applying a rebase, we normally go from an older base commit to a newer one.
A ---- B ---- C ---- D -origin/master
\
E -branch
In this case, `git rebase D` would certainly work as expected. However,
writing `git rebase B` would NOT get us to the following state:
A ---- B ---- C ---- D -origin/master
\
E' -branch
In fact, it would have no effect.
This article http://matthew-brett.github.io/pydagogue/rebase_without_tears.html
explains the general invocation as
> `git rebase --onto <graft-point> <exclude-from> <include-from>`
> If you don’t specify --onto, <graft-point> defaults to <exclude-from>
So what's happening is, by writing `git rebase B` we're rebasing onto B,
excluding commits that are in B. Commit C is not "in" B so it is kept and we're
back to the starting point.
So I suggest to change the invocation to `git rebase --onto B origin/master`,
which rebases onto B, excluding commits that are in origin/master. This works
more generally and allows rebasing "backwards".
Bug: None
Change-Id: I68e4d805811530b585550bc75099354fef4e9c15
Reviewed-on: https://chromium-review.googlesource.com/904004
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Oleh Prypin <oprypin@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).
Please review the expectation changes, and LGTM+CQ.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
recipe_engine:
https://crrev.com/77667b5ba38c4d205a743d04eb625ac068bdd78e Always use Gerrit for tryserver recipe tests (agable@chromium.org)
R=dnj@chromium.org, martiniss@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I2df24323974ae0d3b3b0524240b64be5740977c6
Reviewed-on: https://chromium-review.googlesource.com/836760
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This partially reverts commit d51ed57edb.
Reason for revert:
New git client for windows was rolled including fix for slow `git fetch`.
I guess smaller pack limit causes frequent bootstrap taking 2~3 minutes longer than
the case it does not happen.
Let me see what happen if we increase pack limit 9 -> 30.
I will increase this to 50 if this won't cause regression again.
Original change's description:
> git_cache: lower max num of .pack files before re-bootstrap on Win.
>
> It used to be 50, I think ~9 gives best results for Chromium on Win:
> on golo VM, it takes <4 minutes to re-boostrap + git fetch small
> delta, assuming zipped git checkout for bootstrap is fresh (~1day).
>
> For other repos, which are significantly smaller, this change should
> have minor effect if at all.
>
> Test: I tested this using `led` tool on Win7 machines running LUCI
> stack extensively. For example,
>
> * https://ci.chromium.org/swarming/task/3a0102e8c8657410
> shows case with few .pack files, hence just 1 fetch
>
> * https://ci.chromium.org/swarming/task/3a010282f9fd8010
> shows case with 39 .pack files and so bootstrapping + fetch.
> If you look at prior tasks on the same VM, you'd find this:
> https://ci.chromium.org/swarming/task/39ffe843d01ed010
> which spent 8 minutes doing 1 incremental fetch with 39 .pack
> files.
>
> **Troopers/Sheriffs**: This change is safe to revert.
> However, beware that you should also at the same time revert the recipe
> roll of this CL to the repo, in which the failed builder's recipe is
> located, typically `chromium/tools/build`.
>
> Bug: 749709
> Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
> Reviewed-on: https://chromium-review.googlesource.com/787174
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 749709
Change-Id: I3052abe4a9b53277a60c0791a85355e7a0bbdf8f
Reviewed-on: https://chromium-review.googlesource.com/823544
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This will be used by various things in build and build_internal which
"need" a depot_tools on PATH (though I suspect most of those should be
refactored to use the depot_tools in chromium/src.git).
R=tandrii@chromium.org
Bug: 789808
Change-Id: I5d0590ccdc5482bd8e3048ad46a39b5aedacbddb
Reviewed-on: https://chromium-review.googlesource.com/828465
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
$ git checkout <name>
may be interpreted by git as checking out a file named <name>
if <name> has slashes. Thus,
$ git checkout infra/config
ends up checking out infra/config directory of the (typically)
master branch that was checked out before. To avoid confusion,
we have to add '--' separator between branch name and file names:
$ git checkout <name> --
This in turns allows to run CQ-based presubmit on infra/config branches.
R=nodir@chromium.org
Bug: 790738
Change-Id: I6ce31a8f0fbd66fd59ac7c2ea9cccd3ff97d1f0e
Reviewed-on: https://chromium-review.googlesource.com/802136
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
bot_update and gclient invokes git cache with --ignore-locks.
This should also reduce noisiness of bot_update stdout.
R=iannucci@chromium.org
Bug:
Change-Id: I6baf9e5665d244d08aab58a7d3327e6c3204b2b5
Reviewed-on: https://chromium-review.googlesource.com/791536
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Tested end-to-end, for example
https://ci.chromium.org/swarming/task/3a0147207378b910
which contains:
src/media/cdm/api (Elapsed: 0:00:01)
----------------------------------------
[0:00:00] Started.
_____ src\media\cdm\api at ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237
[0:00:00] running "git cat-file -e ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237" in "e:\b\s\w\ir\cache\git\chromium.googlesource.com-chromium-cdm"
skipping mirror update, it has rev=ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237 already
thereby saving on needless git fetch (~40s in glcient sync on win trybots),
and reducing the rate of .pack file accumulation inside cache directories.
Risks: silently broken recipes which run gclient sync (or worse, bot_update)
as a means of fetching latest commits in all repos of a solution. I think
the benefit of faster bot_update in chromium CQ is worth the potential risk.
PSA: https://groups.google.com/a/chromium.org/d/msg/infra-dev/UYLdBwAXm1Y/OV9QB6JnBQAJ
Bug: 749709
Change-Id: I7a9e8ab82a5e2b848e450f19a798ac18a0b5e201
Reviewed-on: https://chromium-review.googlesource.com/787331
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Before, bot_update would print after command:
Succeeded (Failed) in 0.1 mins
but figuring out which command takes a lot of scrolling/searching
for previous
Running <cmd with args>
This CL fixes that:
Succeeded (Failed) in 0.1 mins <cmd with args>
R=machenbach@chromium.org, maruel@chromium.org
TEST=end-to-end with led tool, but only on LUCI stack.
Bug: 749709
Change-Id: Id3031a12b9013ab136ea03cdf6d325930722e50f
Reviewed-on: https://chromium-review.googlesource.com/787175
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
It used to be 50, I think ~9 gives best results for Chromium on Win:
on golo VM, it takes <4 minutes to re-boostrap + git fetch small
delta, assuming zipped git checkout for bootstrap is fresh (~1day).
For other repos, which are significantly smaller, this change should
have minor effect if at all.
Test: I tested this using `led` tool on Win7 machines running LUCI
stack extensively. For example,
* https://ci.chromium.org/swarming/task/3a0102e8c8657410
shows case with few .pack files, hence just 1 fetch
* https://ci.chromium.org/swarming/task/3a010282f9fd8010
shows case with 39 .pack files and so bootstrapping + fetch.
If you look at prior tasks on the same VM, you'd find this:
https://ci.chromium.org/swarming/task/39ffe843d01ed010
which spent 8 minutes doing 1 incremental fetch with 39 .pack
files.
**Troopers/Sheriffs**: This change is safe to revert.
However, beware that you should also at the same time revert the recipe
roll of this CL to the repo, in which the failed builder's recipe is
located, typically `chromium/tools/build`.
Bug: 749709
Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
Reviewed-on: https://chromium-review.googlesource.com/787174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This is a reland of 331a276b54
Original change's description:
> bot_update: make --output_json required in script invocation.
>
> It's already set by recipe, so this should have 0 difference in prod.
>
> R=iannucci@chromium.org
>
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Change-Id: I165da9650df6b5e8d382aafc30914eff9a524c56
Reviewed-on: https://chromium-review.googlesource.com/777820
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
This reverts commit 331a276b54.
Reason for revert: Looks like this is causing bots to fail on the waterfall
Example:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F60913%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout
Original change's description:
> bot_update: make --output_json required in script invocation.
>
> It's already set by recipe, so this should have 0 difference in prod.
>
> R=iannucci@chromium.org
>
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
TBR=iannucci@chromium.org,tandrii@chromium.org
Change-Id: I65d7665e45dde539f6436ffe8932319e03f96f25
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/777619
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
It's already set by recipe, so this should have 0 difference in prod.
R=iannucci@chromium.org
Bug:
Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
Reviewed-on: https://chromium-review.googlesource.com/769164
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
The test output was incorrect, this matches what it actually should look like.
Bug: 772529
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: release_scripts
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b
Reviewed-on: https://chromium-review.googlesource.com/751694
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit 47b67c426b.
Reason for revert: Bug in bot_update.py resource, breaks recipe roller.
Original change's description:
> Reland "bot_update recipe: Upload source manifest"
>
> This reverts commit c3d1208d5c.
>
> Also:
> * Instead of replacing "manifest", just add a new "source_manifest"
> to the output JSON. This allow transition without breakage.
> * Change the test api so test for recipe output changes.
>
> The plan is to land this first, switch all downstream to "source_manifest",
> and then remove the original "manifest" key.
>
> Bug: 772529,776299
> Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: release_scripts
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Reviewed-on: https://chromium-review.googlesource.com/731378
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>
TBR=iannucci@chromium.org,hinoka@chromium.org
Change-Id: I7a4ee904075e8b75b8a47f9ef0cd8a633af85a9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529, 776299
Reviewed-on: https://chromium-review.googlesource.com/748312
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This reverts commit c3d1208d5c.
Also:
* Instead of replacing "manifest", just add a new "source_manifest"
to the output JSON. This allow transition without breakage.
* Change the test api so test for recipe output changes.
The plan is to land this first, switch all downstream to "source_manifest",
and then remove the original "manifest" key.
Bug: 772529,776299
Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/731378
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
If the revision has not yet propagated, then we never retry, because the amount of time to fetch HEAD already exceeds the deadline.
Bug: 771007
Change-Id: I91ac30b2b53b4f75729357b3298c52856a4d350e
Reviewed-on: https://chromium-review.googlesource.com/729328
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
These aren't in use, and the original problem they were
meant to solve has been solved at the gclient.py layer
using resource locking:
https://codereview.chromium.org/2049583003
Bug: 773008
Change-Id: I6609f39d7f15604e0bb3d742a41c4f9fec87a57a
Reviewed-on: https://chromium-review.googlesource.com/707728
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This is a step away from patch_project, which was a holdover
from Rietveld. Instead of mapping "projects" (which are
poorly-defined and not guaranteed to be unique) to subpaths,
instead map repository urls (which are at least unique).
Eventually we may be able to compute this directly from DEPS
instead of hardcoding the mapping here, but at least this is
a step in the right direction.
Bug: 765633
Change-Id: Idd65984fc6edefcbedb0438d38c2338b10b7e8e5
Reviewed-on: https://chromium-review.googlesource.com/690776
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Bug: 765633
Change-Id: I8f2a9ba807fbe176695a7d44467f3948eea07fda
Reviewed-on: https://chromium-review.googlesource.com/668446
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Print the Python version during "bot_update". This is useful for
visually auditing the environment in which things are running.
BUG=None
TEST=None
R=iannucci@chromium.org
Change-Id: I92be8a888e6f1e5c17205665dff36378e641cd81
Reviewed-on: https://chromium-review.googlesource.com/636450
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
The only user of this method needs to apply two patches from two
different repos in a try job. Extending the method to handle this.
BUG=628259
R=agable@chromium.org
Change-Id: Ia119780d1323add0094631e5c291b0e7a573e6e5
Reviewed-on: https://chromium-review.googlesource.com/588008
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This was removed in downstream recipes after it was turned on by default.
Bug: 740456
Change-Id: Idae1905af97333970c8b31508634f8b71550c6f9
Reviewed-on: https://chromium-review.googlesource.com/574030
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Robert Iannucci
Tom Anderson
Andrii Shyshkalov
Oleh Prypin
Takuto Ikuta
Aaron Gable
recipe-roller
Robert Flack
Ryan Tseng
Ryan Tseng
Jao-ke Chin-Lee
Michael Achenbach
Vadim Shtayura
Edward Lemur
Dan Jacques
John Budorick
Sergey Berezin