This is a reland of 9219d35688
The original was reverted due to a typo (core,quotePath instead
of core.quotePath). This version is fixed.
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
>
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I28d2260948aaf63bd865888c2f60e4cdee9aea48
Reviewed-on: https://chromium-review.googlesource.com/822990
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This reverts commit 9219d35688.
Reason for revert: unfortunately this says "core,quotePath" and since it includes recipe changes, we need something that the roller can munch on :(
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
TBR=maruel@chromium.org,agable@chromium.org
Change-Id: I226388f19024403240a1443eb2b878b9293220e1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 792302
Reviewed-on: https://chromium-review.googlesource.com/821671
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
This prevents git from putting quotes around some file names
(those that have astral-plane characters) and not around others.
R=maruel
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
Reviewed-on: https://chromium-review.googlesource.com/815454
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
$ git checkout <name>
may be interpreted by git as checking out a file named <name>
if <name> has slashes. Thus,
$ git checkout infra/config
ends up checking out infra/config directory of the (typically)
master branch that was checked out before. To avoid confusion,
we have to add '--' separator between branch name and file names:
$ git checkout <name> --
This in turns allows to run CQ-based presubmit on infra/config branches.
R=nodir@chromium.org
Bug: 790738
Change-Id: I6ce31a8f0fbd66fd59ac7c2ea9cccd3ff97d1f0e
Reviewed-on: https://chromium-review.googlesource.com/802136
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
bot_update and gclient invokes git cache with --ignore-locks.
This should also reduce noisiness of bot_update stdout.
R=iannucci@chromium.org
Bug:
Change-Id: I6baf9e5665d244d08aab58a7d3327e6c3204b2b5
Reviewed-on: https://chromium-review.googlesource.com/791536
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Tested end-to-end, for example
https://ci.chromium.org/swarming/task/3a0147207378b910
which contains:
src/media/cdm/api (Elapsed: 0:00:01)
----------------------------------------
[0:00:00] Started.
_____ src\media\cdm\api at ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237
[0:00:00] running "git cat-file -e ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237" in "e:\b\s\w\ir\cache\git\chromium.googlesource.com-chromium-cdm"
skipping mirror update, it has rev=ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237 already
thereby saving on needless git fetch (~40s in glcient sync on win trybots),
and reducing the rate of .pack file accumulation inside cache directories.
Risks: silently broken recipes which run gclient sync (or worse, bot_update)
as a means of fetching latest commits in all repos of a solution. I think
the benefit of faster bot_update in chromium CQ is worth the potential risk.
PSA: https://groups.google.com/a/chromium.org/d/msg/infra-dev/UYLdBwAXm1Y/OV9QB6JnBQAJ
Bug: 749709
Change-Id: I7a9e8ab82a5e2b848e450f19a798ac18a0b5e201
Reviewed-on: https://chromium-review.googlesource.com/787331
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Before, bot_update would print after command:
Succeeded (Failed) in 0.1 mins
but figuring out which command takes a lot of scrolling/searching
for previous
Running <cmd with args>
This CL fixes that:
Succeeded (Failed) in 0.1 mins <cmd with args>
R=machenbach@chromium.org, maruel@chromium.org
TEST=end-to-end with led tool, but only on LUCI stack.
Bug: 749709
Change-Id: Id3031a12b9013ab136ea03cdf6d325930722e50f
Reviewed-on: https://chromium-review.googlesource.com/787175
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
It used to be 50, I think ~9 gives best results for Chromium on Win:
on golo VM, it takes <4 minutes to re-boostrap + git fetch small
delta, assuming zipped git checkout for bootstrap is fresh (~1day).
For other repos, which are significantly smaller, this change should
have minor effect if at all.
Test: I tested this using `led` tool on Win7 machines running LUCI
stack extensively. For example,
* https://ci.chromium.org/swarming/task/3a0102e8c8657410
shows case with few .pack files, hence just 1 fetch
* https://ci.chromium.org/swarming/task/3a010282f9fd8010
shows case with 39 .pack files and so bootstrapping + fetch.
If you look at prior tasks on the same VM, you'd find this:
https://ci.chromium.org/swarming/task/39ffe843d01ed010
which spent 8 minutes doing 1 incremental fetch with 39 .pack
files.
**Troopers/Sheriffs**: This change is safe to revert.
However, beware that you should also at the same time revert the recipe
roll of this CL to the repo, in which the failed builder's recipe is
located, typically `chromium/tools/build`.
Bug: 749709
Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
Reviewed-on: https://chromium-review.googlesource.com/787174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This is a reland of 331a276b54
Original change's description:
> bot_update: make --output_json required in script invocation.
>
> It's already set by recipe, so this should have 0 difference in prod.
>
> R=iannucci@chromium.org
>
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Change-Id: I165da9650df6b5e8d382aafc30914eff9a524c56
Reviewed-on: https://chromium-review.googlesource.com/777820
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
This reverts commit 331a276b54.
Reason for revert: Looks like this is causing bots to fail on the waterfall
Example:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F60913%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout
Original change's description:
> bot_update: make --output_json required in script invocation.
>
> It's already set by recipe, so this should have 0 difference in prod.
>
> R=iannucci@chromium.org
>
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
TBR=iannucci@chromium.org,tandrii@chromium.org
Change-Id: I65d7665e45dde539f6436ffe8932319e03f96f25
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/777619
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
It's already set by recipe, so this should have 0 difference in prod.
R=iannucci@chromium.org
Bug:
Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
Reviewed-on: https://chromium-review.googlesource.com/769164
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Bug:
Change-Id: I031f5787a87a4bddbd3492912814476daf2c9b4f
Reviewed-on: https://chromium-review.googlesource.com/738118
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
The test output was incorrect, this matches what it actually should look like.
Bug: 772529
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: release_scripts
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b
Reviewed-on: https://chromium-review.googlesource.com/751694
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit 47b67c426b.
Reason for revert: Bug in bot_update.py resource, breaks recipe roller.
Original change's description:
> Reland "bot_update recipe: Upload source manifest"
>
> This reverts commit c3d1208d5c.
>
> Also:
> * Instead of replacing "manifest", just add a new "source_manifest"
> to the output JSON. This allow transition without breakage.
> * Change the test api so test for recipe output changes.
>
> The plan is to land this first, switch all downstream to "source_manifest",
> and then remove the original "manifest" key.
>
> Bug: 772529,776299
> Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: release_scripts
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Reviewed-on: https://chromium-review.googlesource.com/731378
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>
TBR=iannucci@chromium.org,hinoka@chromium.org
Change-Id: I7a4ee904075e8b75b8a47f9ef0cd8a633af85a9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529, 776299
Reviewed-on: https://chromium-review.googlesource.com/748312
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This reverts commit c3d1208d5c.
Also:
* Instead of replacing "manifest", just add a new "source_manifest"
to the output JSON. This allow transition without breakage.
* Change the test api so test for recipe output changes.
The plan is to land this first, switch all downstream to "source_manifest",
and then remove the original "manifest" key.
Bug: 772529,776299
Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/731378
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
If the revision has not yet propagated, then we never retry, because the amount of time to fetch HEAD already exceeds the deadline.
Bug: 771007
Change-Id: I91ac30b2b53b4f75729357b3298c52856a4d350e
Reviewed-on: https://chromium-review.googlesource.com/729328
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This fixes an issue introduced by
crrev.com/b40a45149a73a6b00e92fd1d2df4c0b1543d0005 where a 404 result
might return None contents.
R=agable@google.com, phajdan@google.com
Bug:774355
Change-Id: I993a029adb8d92cb27e043cd3d9b17cc8aee2432
Reviewed-on: https://chromium-review.googlesource.com/717937
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
This is needed in the release recipes because there are times when we
want to check that a file is _not_ there yet (i.e. 404 response).
R=agable@chromium.org, dnj@chromium.org
Change-Id: Iedf646d6c73ebf898e90afd2009840b1e5c5b1ab
Reviewed-on: https://chromium-review.googlesource.com/709916
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
These aren't in use, and the original problem they were
meant to solve has been solved at the gclient.py layer
using resource locking:
https://codereview.chromium.org/2049583003
Bug: 773008
Change-Id: I6609f39d7f15604e0bb3d742a41c4f9fec87a57a
Reviewed-on: https://chromium-review.googlesource.com/707728
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This is supported by gclient and has the effect of treating the solution
as a purely local, standalone DEPS (i.e. not as a source repo to
checkout or process in any way). It also prevents the solution from
being added to the output of 'gclient flatten'.
R=agable@chromium.org, phajdan.jr@chromium.org
Change-Id: Ie11246ec7a74097b96ea6a26983a63b3fb3123a5
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/698271
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This is a step away from patch_project, which was a holdover
from Rietveld. Instead of mapping "projects" (which are
poorly-defined and not guaranteed to be unique) to subpaths,
instead map repository urls (which are at least unique).
Eventually we may be able to compute this directly from DEPS
instead of hardcoding the mapping here, but at least this is
a step in the right direction.
Bug: 765633
Change-Id: Idd65984fc6edefcbedb0438d38c2338b10b7e8e5
Reviewed-on: https://chromium-review.googlesource.com/690776
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This arg used to be used in a 'sync' call that was removed in
crrev.com/7090f652d180eeff2663e3e3091d5db14123284c, but it was never
applied to the remaining 'sync' call. This removes the arg and leaves
"--with_branch_heads" (and adds "--with_tags") as default sync flags.
Change-Id: Ice5a592d95a0b09dafddab24769f5edd8835a1c9
Recipe-Manual-Change: build_limited_scripts_slave
Recipe-Manual-Change: build
Reviewed-on: https://chromium-review.googlesource.com/677754
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
Goal is to eventually add a presubmit trybot for skia_buildbot
repository.
Similar to one that was added in
https://codereview.chromium.org/1141583002
Bug: skia:7049
Change-Id: I95a8e2b452a44a50dc177d520af4ce95129db6da
Reviewed-on: https://chromium-review.googlesource.com/671410
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Ravi Mistry <rmistry@chromium.org>
Bug: 765633
Change-Id: I8f2a9ba807fbe176695a7d44467f3948eea07fda
Reviewed-on: https://chromium-review.googlesource.com/668446
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Henrik Kjellander <kjellander@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Print the Python version during "bot_update". This is useful for
visually auditing the environment in which things are running.
BUG=None
TEST=None
R=iannucci@chromium.org
Change-Id: I92be8a888e6f1e5c17205665dff36378e641cd81
Reviewed-on: https://chromium-review.googlesource.com/636450
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
R=agable@chromium.org, dnj@chromium.org, vadimsh@chromium.org
Bug: 726507
Change-Id: I8cc5619a5fe93c76ffff32a8915c8b9bf3e077d3
Reviewed-on: https://chromium-review.googlesource.com/607649
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Aaron Gable
Robbie Iannucci
Andrii Shyshkalov
Takuto Ikuta
Robert Flack
William Hesse
Ryan Tseng
Ryan Tseng
Jao-ke Chin-Lee
Michael Achenbach
Michael Moss
Vadim Shtayura
Emma
Ravi Mistry
Edward Lemur
Dan Jacques
John Budorick
Robert Iannucci