Commit Graph

335 Commits (83aafc979fa6536d4d38b18b51e972bdeccdf1d3)

Author SHA1 Message Date
Scott Graham e660e3fb9a Support target_arch for WindowsSDK
Bug: chromium:892712
Change-Id: I68e1822a9a7205aaab3836be3f57a2f1feb1c9a6
Reviewed-on: https://chromium-review.googlesource.com/c/1289231
Commit-Queue: Scott Graham <scottmg@chromium.org>
Auto-Submit: Scott Graham <scottmg@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
John Budorick 1333933200 bot_update: stop failing with a KeyError in the recipe on failure.
This has been low-key driving me bananas for a while, and I've seen
it confuse sheriffs etc many times.

Change-Id: If46ab55821447a3a70a3f0ea774c017f29fc7400
Reviewed-on: https://chromium-review.googlesource.com/c/1286819
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Ryan Tseng 2fddb95698 [git recipe_module] Add tags flag for fetch
Some projects that use the git module (eg flutter) require tags.

Bug:  870558
Change-Id: Idae50ef5c8a092e5f0109b6c14c562268a2ad236
Reviewed-on: https://chromium-review.googlesource.com/c/1274245
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
7 years ago
Robert Iannucci 85af45334e [windows_sdk] Default to cache/windows_sdk.
This should allow better integration with buildbucket named cache
support. Since this was already parameterized it shouldn't actually
break anything; previously the directory would be deleted on every run,
now it will still be deleted on every run (but we can add a buildbucket
named cache entry for it to preserve it).

R=tandrii@chromium.org, vadimsh@chromium.org

Bug: 877775
Change-Id: I1184fb59a3ed18bfda5fcb288903201b01527096
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/1263861
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
7 years ago
Edward Lemur 0c3bd49069 bot_update: Don't use apply_patch_on_gclient.
It has been True by default for a while, and there is no need to override it.

Bug: 891917
Change-Id: I2598a2230b0ea38a647a533757331c541b871971
Reviewed-on: https://chromium-review.googlesource.com/c/1260057
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Nodir Turakulov 92df4e63ab Revert "[gclient] Use canonical URLs in solutions"
This reverts commit 7e301cfffe.

Reason for revert: internal builds failed, e.g. bugdroid

Original change's description:
> [gclient] Use canonical URLs in solutions
> 
> Strip ".git" suffix from all solution URLs.
> 
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: release_scripts
> Bug: 877161
> Change-Id: I0cb9cc512fdd9cc5e53ade6bc1d725dc84f7563d
> Reviewed-on: https://chromium-review.googlesource.com/1237332
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>

TBR=nodir@chromium.org,tandrii@chromium.org,martiniss@chromium.org,jbudorick@chromium.org

Change-Id: Ie24e10844ceb89071c9e665f2aa7dde9f9bf6e6c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 877161
Reviewed-on: https://chromium-review.googlesource.com/1237423
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov 7e301cfffe [gclient] Use canonical URLs in solutions
Strip ".git" suffix from all solution URLs.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: release_scripts
Bug: 877161
Change-Id: I0cb9cc512fdd9cc5e53ade6bc1d725dc84f7563d
Reviewed-on: https://chromium-review.googlesource.com/1237332
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov f837545214 [gclient] Remove patch_projects
It was superseded by repo_path_map.

Bug: 694348
Change-Id: If6b4790290bf2573f2a9367334decf064d382f52
Reviewed-on: https://chromium-review.googlesource.com/1232875
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Robert Iannucci 77900be4e7 [windows_sdk] Allow taskkill mspdbsrv to fail.
It looks like PNaCl actually depends on MSVC's presance, but doesn't
actually run any compile steps with it (instead using clang). That means
that mspdbsrv never actually runs during their build, and then the
taskkill fails.

R=hinoka@chromium.org, tandrii@chromium.org

Bug: 861512
Change-Id: I004d28f198224adaf16b1d3f14401b0d2a7d700b
Reviewed-on: https://chromium-review.googlesource.com/1232885
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Nodir Turakulov 79c651330b [gclient] Remove is_blink_mode
Not used

R=tandrii@chromium.org

Bug: 694348
Change-Id: I3188881b1f92df1cbfdbf3aafe309abb9488d9b2
Reviewed-on: https://chromium-review.googlesource.com/1232874
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Nodir Turakulov 15b586a493 Remove PATCH_PROJECT
Remove PATCH_PROJECT from gclient config. It is not used.

R=tandrii@chromium.org

Bug: 694348
Change-Id: I56e4aefbe5e9b23296553bb61f878d6426197fa7
Reviewed-on: https://chromium-review.googlesource.com/1232873
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Nodir Turakulov 53e28ecb9f [gclient] Remove calculate_patch_root
It is not used. get_gerrit_patch_root is used instead.

Bug: 694348
Change-Id: Icc717e38cbcaa8050c8ce8a364202b81d7f9cf2c
Reviewed-on: https://chromium-review.googlesource.com/1231282
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov 78eed01c0b [bot_update] Use get_gerrit_patch_root
Use get_gerrit_patch_root in bot_update.ensure_checkout.

This fixes some tests expectations that expected CI builds to pass --patch_root
flag.

Also unify "root" and "patch_root" variables in ensure_checkout. Simpler.

This CL requires nontrivial rolls in all downstream packages because all
packages have incorrect expectations and/or tests.
In particular, this removes --patch_root flag from non-tryjob test
cases. Some tests are supposed to test tryjob scenario, but they use
properties.generic instead of properties.tryserver (or buildbucket.try_build)

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skiabuildbot
Bug: 694348
Change-Id: Ibc594fa42e3b4d2823672b41dda2d97154b10d38
Reviewed-on: https://chromium-review.googlesource.com/1231280
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov dd2ce986ce [gclient] Update set_patch_project_revision
Update set_patch_project_revision to use repo_path_map instead of
patch_projects.
Also rename to set_patch_repo_revision.

Bug: 694348
Change-Id: Iba31f4910f8282160000b8fcef197052597530a4
Reviewed-on: https://chromium-review.googlesource.com/1226107
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Nodir Turakulov 33f71ae4d0 [bot_update] Update examples to use buildbucket module
Update bot_update full.py example to use buildbucket module.
Also remove buidlbucket example.

Bug: 694348
Change-Id: Ib784583e0ded5ac95fabe99bc39a53f4646e80fc
Reviewed-on: https://chromium-review.googlesource.com/1231281
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov 7b54572eb0 [gclient] Fix typo in get_gerrit_patch_root
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1217670/17
used solution instead of solution name.

TBR=tandrii@chromium.org

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Bug: 694348
Change-Id: I829cba9223d73383791b018f9a23598443e8fa2c
Reviewed-on: https://chromium-review.googlesource.com/1231278
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov 0f322bfcf5 [bot_update] Add gclient.get_gerrit_patch_root
Add a shortcut for getting local path to the patch repo.

In bot_update.api._destination_branch, start using it instead of reading
gclient config directly. This removes one more patch_project usage.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Bug: 694348
Change-Id: If05371d0f6d11be6803ffa8392e029f5860dda25
Reviewed-on: https://chromium-review.googlesource.com/1217670
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov ddda0b5b8a Fetch current Gerrit CL info in tryserver
Add tryserver.api.gerrit_change_info that returns detailed info about the
current Gerrit CL. It makes a Gerrit RPC in the background on the first read.
This step will be present in all tryjobs.

This mechanism can be used for extra details about the current CL.
This CL starts to use this in bot_update for gerrit_ref, and stops using
patch_ref property, and also update bot_update to use when the destination
branch is needed.

Also this CL removes no longer used properties in bot_update,
namely most of patch_* properties.

build.git manual roll: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1217626

Recipe-Manual-Change: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: release_scripts
R=tandrii@chromium.org
Bug: 694348
Change-Id: I945d061b74278c3d1d227578abe16a64cdaca647
Reviewed-on: https://chromium-review.googlesource.com/1213557
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Nodir Turakulov 11bc399153 [bot_update] add ignore_input_commit parameter
CrOS chromium-pfq builder uses bot_update, but checks out Chromium on its own,
effectively ignoring input commit in bot_update.
To resolve a P0, add ignore_input_commit parameter.
Long term, bot_update API must be redesigned.

TBR=tandrii@chromium.org
Bug: 882094
Change-Id: I4f43348e3d5f81bf603dd2c110829bce58acad16
Reviewed-on: https://chromium-review.googlesource.com/1216787
Reviewed-by: Lann Martin <lannm@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov d91f0adca2 Reland "[bot_update] Stop using repository properties"
This is a reland of 4d2b901804
verbatim. The problem was in the buildbucket recipe module, see
https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1215002#message-2351f1089af71655dbaff576aa4d11f1dedc800b

Original change's description:
> [bot_update] Stop using repository properties
>
> Stop using repository and patch_repository_url properties.
> Use self.m.buildbucket.build.input.gerrit_changes[0].{host, project}
> instead.
>
> R=ehmaldonado@chromium.org, tandrii@chromium.org
>
> Recipe-Nontrivial-Roll: build
> Bug: 877161, 694348
> Change-Id: Ideb069c6c8f8b6e6cfb5217f8a04ea18f16d3056
> Reviewed-on: https://chromium-review.googlesource.com/1208368
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: skia
Bug: 877161, 694348
Change-Id: I0a10ca7814a8f5bdfd30a491e36fb7261cdaf225
Reviewed-on: https://chromium-review.googlesource.com/1214705
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov f022f91453 Revert "[bot_update] Stop using repository properties"
This reverts commit 4d2b901804.

Reason for revert: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1214622

Original change's description:
> [bot_update] Stop using repository properties
> 
> Stop using repository and patch_repository_url properties.
> Use self.m.buildbucket.build.input.gerrit_changes[0].{host, project}
> instead.
> 
> R=​ehmaldonado@chromium.org, tandrii@chromium.org
> 
> Recipe-Nontrivial-Roll: build
> Bug: 877161, 694348
> Change-Id: Ideb069c6c8f8b6e6cfb5217f8a04ea18f16d3056
> Reviewed-on: https://chromium-review.googlesource.com/1208368
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

TBR=nodir@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org

Change-Id: I825242124382e9f1789d4a3da0a3bdd2a5fcab37
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 877161, 694348
Reviewed-on: https://chromium-review.googlesource.com/1214703
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov 4d2b901804 [bot_update] Stop using repository properties
Stop using repository and patch_repository_url properties.
Use self.m.buildbucket.build.input.gerrit_changes[0].{host, project}
instead.

R=ehmaldonado@chromium.org, tandrii@chromium.org

Recipe-Nontrivial-Roll: build
Bug: 877161, 694348
Change-Id: Ideb069c6c8f8b6e6cfb5217f8a04ea18f16d3056
Reviewed-on: https://chromium-review.googlesource.com/1208368
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov f3d9294ba4 [bot_update] Pass git_repo param in test
Pass git_repo param when calling buildbucket.test_api.try_build.

TBR=hinoka@chromium.org
Change-Id: Ic4a3d270ef3477cb21e4e02d8a22caf7996757b3
Reviewed-on: https://chromium-review.googlesource.com/1213555
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov bf6feefa8a Update tryserver to use buildbucket.build.input.gerrit_changes
tryserver module's purpose is to provide high level API for patches.
Update it to use buildbucket.build.input.gerrit_changes[0].
Add helper properties for gerrit repo URL and gerrit ref.

This will be used by both bot_update and gclient.

R=hinoka@chromium.org, tandrii@chromium.org

Bug: 694348
Change-Id: I7eb416e75737310404c4354fa680fd3499eddaa7
Reviewed-on: https://chromium-review.googlesource.com/1211653
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Nodir Turakulov 60e7f52377 [bot_update] Stop using revision and parent_got_revision properties
Use self.m.buildbucket.build.input.gitiles_commit.id instead.
It encapsulates revision and parent_got_revision properties.

Also, honor host and project of the specified commit.
This enables overriding non-first gclient solution at the build-level
for arbitrary buildbucket builds.


Bug: 877161
Change-Id: I3f92f04ca84a1d30cbbb2670f62bea2bd9ee5932
Reviewed-on: https://chromium-review.googlesource.com/1200066
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov 4e3920bef0 [gclient] Start actively using repo_path_map
Use repo_path_map in all places where patch_projects is currently used.
Add gclient.get_repo_path function to resolve path to a repo given its URL.

infra.git requires a non-trivial roll because of incorrect expectations.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Bug: 877161
Change-Id: Iaea0a3f14609bf0c18d4ecdf8564153b8052584d
Reviewed-on: https://chromium-review.googlesource.com/1208363
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov 9af7d107de Revert "bot_update: Specify the target branch for patch_refs."
This reverts commit 01f0c698fb.

Reason for revert: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1211422

Original change's description:
> bot_update: Specify the target branch for patch_refs.
> 
> Bug: 870279
> Recipe-Manual-Change: build
> Recipe-Manual-Change: infra
> Recipe-Manual-Change: build_limited_scripts_slave
> Recipe-Manual-Change: skia
> Recipe-Manual-Change: release_scripts
> Change-Id: I58110ff8c5a6ca6906bbbf3eafa09dcbaf39f777
> Reviewed-on: https://chromium-review.googlesource.com/1174988
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

TBR=tandrii@chromium.org,ehmaldonado@chromium.org,jbudorick@chromium.org

Change-Id: Ic78a264ccf5e2ca15452659e17d41da264560eb8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 870279
Reviewed-on: https://chromium-review.googlesource.com/1211462
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Corentin Wallez 1e591dca21 Add Dawn got_revision_mapping to glient config
BUG=875245

Change-Id: I50425aef624d00c1aa6fd7c18244fea128c15842
Reviewed-on: https://chromium-review.googlesource.com/1209643
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lemur 01f0c698fb bot_update: Specify the target branch for patch_refs.
Bug: 870279
Recipe-Manual-Change: build
Recipe-Manual-Change: infra
Recipe-Manual-Change: build_limited_scripts_slave
Recipe-Manual-Change: skia
Recipe-Manual-Change: release_scripts
Change-Id: I58110ff8c5a6ca6906bbbf3eafa09dcbaf39f777
Reviewed-on: https://chromium-review.googlesource.com/1174988
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov f215ae6ed0 [gitiles] Add parse_repo_url function
Add a function to parse a Gitiles repo URL supporting various edge cases.

R=tandrii@chromium.org

Bug: 877161
Change-Id: Ib4d6a28a2ace2e594f1a4ca31460a446fc6fa052
Reviewed-on: https://chromium-review.googlesource.com/1204914
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Nodir Turakulov efc95d6b9a [bot_update] Untie revision and repository properties
repository and revision have less to do with each other than it seems:
revision is for checkout and the part of code that reads revision does not
read repository.
repository property is used to determine what to apply the patchset to.

When initializing their values from buildbucket.build.input, do it separately.
Also, initialize repository from Gerrit changes, not Gitiles commit.

See also
https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1199854
https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1199856
Bug: 877161, 694348
Change-Id: I4907a31ce055a3526b15f1a3c8e8f15c2c831437
Reviewed-on: https://chromium-review.googlesource.com/1200065
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
7 years ago
Nodir Turakulov f20c5a4a3c [bot_update] Fix initialization from buildbucket
When checking if buildbucket.build_input.gitiles_commit exists, use
HasField protobuf method, as opposed to truth check. A protobuf message
field is always true.

Bug: 877161
Change-Id: Ie4a7e22b985776441a74283b09646e8837c326f1
Reviewed-on: https://chromium-review.googlesource.com/1196930
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
John Budorick b16da6a2a9 bot_update: set the push URL to the actual remote, not the cache.
Bug: 876734
Change-Id: I5741b526f14d79f7a8eb42808c1eef93434739cb
Reviewed-on: https://chromium-review.googlesource.com/1187368
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Corentin Wallez 29361ca11f Add standalone dawn gclient config
BUG=chromium:875245

Change-Id: Ieed7554c55bab2bc5820cf4ffbbf57b72492b90f
Reviewed-on: https://chromium-review.googlesource.com/1183432
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
7 years ago
Ben Pastene dd765da4df depot_tools: Add header doc w/ a warning to cipd recipe_module.
Change-Id: Id1c0ab59864c0e1043751f8825828f7ae9cf9ac4
Reviewed-on: https://chromium-review.googlesource.com/1180479
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
John Budorick 7de5f08c45 bot_update: remove --gerrit_repo and --gerrit_ref.
Bug: 865882
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Change-Id: I4a04c4c28c7dc63ad282f54604c300bbfe654f11
Reviewed-on: https://chromium-review.googlesource.com/1157687
Commit-Queue: John Budorick <jbudorick@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 2f38dbdd97 Register expect_tests in gclient config.
This is in order to use built-in presubit recipe.

R=maruel

Change-Id: I3e50af7dc3a9ad83f0f358285395794ea599a4e8
Reviewed-on: https://chromium-review.googlesource.com/1176650
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
John Budorick ba883cb5ed bot_update: only fetch explicit refs for the main solution.
Bug: 865882
Change-Id: I27bfbeaf7395a05ce660e9e961550233461acb3d
Reviewed-on: https://chromium-review.googlesource.com/1175857
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Andrii Shyshkalov 52c7211807 Set explicit latest OWNERS for bot_update recipe_module.
R=ehmaldonado@chromium.org

Change-Id: Ic730530bc53632a64fb79a733e0a00ba2460da67
Reviewed-on: https://chromium-review.googlesource.com/1175242
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
John Budorick e94deafd14 bot_update: pull revisions from provided cfg if present.
Bug: 873186
Change-Id: I91649cc5f4ba8f81f982463aa534e77d88f87ac0
Reviewed-on: https://chromium-review.googlesource.com/1175099
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Edward Lemur 6a4e31b488 gclient_scm: Add option to specify the branch the patch is based on.
Also add shortcuts, so we don't have to look through all the branches
if the patch is based on a common branch (i.e. master, infra/config, lkgr)


Bug: 870279
Change-Id: I625a8481dccac9a475b096b926e6fab7efe676b0
Reviewed-on: https://chromium-review.googlesource.com/1161094
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
recipe-roller 284b5e0c60 Roll recipe dependencies (nontrivial).
This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).


Please review the expectation changes, and LGTM+CQ.


More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
recipe_engine:
  https://crrev.com/bd5111dec043a367d69d85b81289e2207aa26f6e [cipd] Clean docs, pylint errors, add pkg-fetch, pkg-deploy. (iannucci@chromium.org)


R=tandrii@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I19d44529f7caa41c9577e57f6a9a35815627c4c3
Reviewed-on: https://chromium-review.googlesource.com/1170243
Commit-Queue: Recipe Roller <recipe-roller@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Robert Iannucci 61110ec711 [osx_sdk] Add mac_toolchain based osx_sdk module.
R=hinoka@chromium.org, nodir@chromium.org, sergeyberezin@chromium.org

Change-Id: I92a6675a7fa81987667edb8c7189c2e827f62732
Reviewed-on: https://chromium-review.googlesource.com/1168652
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Ryan Tseng 735f71d95b [recipe_modules] Copy over windows_sdk from infra.git
Change-Id: Ibc528126b26c4f7b7f2e308c89390878bfcfde86
Reviewed-on: https://chromium-review.googlesource.com/1166207
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
John Budorick 5300616351 bot_update: expose patch_refs as an option to ensure_checkout.
Bug: 865882
Change-Id: Icd143fcb6abe1e880dec46a6d26a84103ab4e6d2
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Reviewed-on: https://chromium-review.googlesource.com/1157514
Commit-Queue: John Budorick <jbudorick@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
John Budorick 3e65878e5e Add --patch_ref to bot_update.
This doesn't expose --patch_ref in the bot_update recipe_module yet,
though.

Bug: 865882
Change-Id: I26abe59d2f6a8a3dd3945cce0bf13b238a7fffd3
Reviewed-on: https://chromium-review.googlesource.com/1151105
Commit-Queue: John Budorick <jbudorick@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lemur 455ee3a61e bot_update: Don't pass reset-fetch-config to gclient
Forgot to remove it in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1135645

No-Presubmit: True
Tbr: agable@chromium.org
Bug: 862547
Change-Id: I926959cd91a89d8e77ae000543b00b667b5eb25f
Reviewed-on: https://chromium-review.googlesource.com/1137362
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur 579c98612b Reset the fetch config in the bots before each run.
If a ref is incorrectly set, it will affect later runs.
For example, [1] caused a bad refspec to be added to the fetch config,
(+refs/branch-heads/refs/branch-heads/6.7:refs/branch-heads/refs/branch-heads/6.7)
which affects all subsequent runs in the same bot: [2], [3].

This makes bot_update clear the fetch config before each run, so it is more
robust against these kind of failures.

[1] https://logs.chromium.org/v/?s=v8%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8942033142534034848%2F%2B%2Fsteps%2Fbot_update%2F0%2Fstdout
[2] https://logs.chromium.org/v/?s=v8%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8941307871443322944%2F%2B%2Fsteps%2Fbot_update%2F0%2Fstdout
[3] https://logs.chromium.org/v/?s=v8%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8941333453153279680%2F%2B%2Fsteps%2Fbot_update%2F0%2Fstdout

Bug: 862547
Change-Id: I2f849c604656e81ebd7377465d287226b8bdea1a
Reviewed-on: https://chromium-review.googlesource.com/1135645
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Edward Lemur 92745a5b0e bot_update: Fix syncing to origin/*.
Skip adding origin if the branch name already has 'origin/'.

Bug: 859703
Change-Id: I0141c04189d00187c202a7885a7c6bb74571e7ea
Reviewed-on: https://chromium-review.googlesource.com/1128390
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lemur 4fb2477d08 bot_update: Fix syncing to branches.
When syncing remote branches, checkout origin/<branch> instead of
<branch>, since there might be a local branch with that name.

Bug: 859703
Change-Id: I9a49b940f53f75a97e5458fdcf578c3b52b26de2
Reviewed-on: https://chromium-review.googlesource.com/1128271
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Michael Achenbach 3c023259d7 [gsutil] Consistently return recipe step result
Obviates need to call api.step.active_result by callers.

Bug: 711249
Change-Id: Ibd3278df01fc8f9d7cc73eacc72d9fc57578808d
Reviewed-on: https://chromium-review.googlesource.com/1127372
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 5484b866dc bot_update: use buildbucket's gitiles_commit as repo/revision input.
Today, whenever LUCI scheduler sets gitiles_commit, a revision property
is also always set, thus this CL will have no effect on today's production.

The goal of CL is to pave the way to avoid this duplication. For starters,
one could start real writing tests with

    + api.buildbucket.ci_build(..., revision='sha1')

without the need to duplicate the same revision as a property. All
recipes which WAI with such input are thus ready to work w/o 'revision'
property. After that, we'll be able to stop the duplication in LUCI scheduler.

R=hinoka

Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Change-Id: I1773b1c3d2a98a9dbd1cfdd9ac2b4fb99745ee99
Reviewed-on: https://chromium-review.googlesource.com/1123713
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
7 years ago
Andrii Shyshkalov 43033409f4 bot_update: remove legacy Gerrit patch property support.
Change-Id: Ie95e10784725176c94a584fa6dadef2318142053
Reviewed-on: https://chromium-review.googlesource.com/1123650
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Sergiy Byelozyorov d38a2da9f1 Use cwd from context instead of kwargs
It is not possible to use cwd argument, since none of the methods called by this
module define cwd explicitly and low-level modules like api.step do not support
it anymore, resulting in a TypeError exception thrown due to unexpected keyword
argument, e.g. see https://ci.chromium.org/swarming/task/3e74a52905a71110. Given
that any users using cwd argument would be broken anyway, it should be safe to
make this change.

Planned usage: https://crrev.com/c/1122874

R=tandrii@chromium.org, iannucci@chromium.org

Bug: 793687
Change-Id: Iadf6bf91d8d41dbc6751a848bb69d7922a206ade
Reviewed-on: https://chromium-review.googlesource.com/1122873
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
7 years ago
Robert Iannucci a4dec94a1a Add download_archive to gitiles module.
R=phosek@chromium.org, tandrii@chromium.org, vadimsh@chromium.org

Bug: 790650
Change-Id: Ia878004d6983dbbad882ec683da2e0db5e727c66
Reviewed-on: https://chromium-review.googlesource.com/1116073
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 77094f7fea Reland of "Wipe out shallow checkout support from bot_update."
Originally-Reviewed-on: https://chromium-review.googlesource.com/1111107
Led-test: https://ci.chromium.org/swarming/task/3e51ed0e31211710?server=chromium-swarm.appspot.com

TBR=hinoka, iannucci

Bug: 855137
Change-Id: I157ef6b9ee2a7ce45e4a474853ae769a82cc4c99
Reviewed-on: https://chromium-review.googlesource.com/1114234
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 612d62d0ca Revert "bot_update: Don't force solutions to be unmanaged."
This reverts commit c48318a35b.

Reason for revert: likely broke ChromeOS

Original change's description:
> bot_update: Don't force solutions to be unmanaged.
> 
> gclient should be the one syncing the solutions to the appropriate
> revisions.
> 
> Bug: 850812, 853032
> Change-Id: Ieefc5661627d4864deb0d4e7053168a99da29d29
> Reviewed-on: https://chromium-review.googlesource.com/1102833
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=hinoka@chromium.org,agable@chromium.org,ehmaldonado@chromium.org

Change-Id: I7e110108a9687d703ea8be39cc83e0888f7813c6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 850812, 853032, 855700
Reviewed-on: https://chromium-review.googlesource.com/1111925
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 565fa6a407 Revert "Wipe out shallow checkout support from bot_update."
This reverts commit 568b7a8998.

Reason for revert: temporary revert to revert previous CL.

Original change's description:
> Wipe out shallow checkout support from bot_update.
> 
> Bug: 855137
> Change-Id: I60b8d69f9ab353c12120d4216e3a12518f99732c
> Reviewed-on: https://chromium-review.googlesource.com/1111107
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Ryan Tseng <hinoka@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,tandrii@chromium.org

Change-Id: Iede67717be448be9bdd7c5dd4895fbab9fd65c66
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 855137, 855700
Reviewed-on: https://chromium-review.googlesource.com/1111924
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 568b7a8998 Wipe out shallow checkout support from bot_update.
Bug: 855137
Change-Id: I60b8d69f9ab353c12120d4216e3a12518f99732c
Reviewed-on: https://chromium-review.googlesource.com/1111107
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
7 years ago
Edward Lemur c48318a35b bot_update: Don't force solutions to be unmanaged.
gclient should be the one syncing the solutions to the appropriate
revisions.

Bug: 850812, 853032
Change-Id: Ieefc5661627d4864deb0d4e7053168a99da29d29
Reviewed-on: https://chromium-review.googlesource.com/1102833
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Andrii Shyshkalov 124365b8ea bot_update: default to non-shallow checkouts.
shallow mode is a special git mode that only checks out a shallow copy
of git so the full history is not downloaded. It is not well supported
by googlesource.com servers because git has to do a giant negotiation
with the server and transmit every single commit that the local checkout
has or doesn't have (instead of just the latest commit of each ref),
which ends up taking more than 10x longer than a non-shallow checkout.

R=ehmaldonado, hinoka, iannucci

Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 855137
Change-Id: I5f9e31f8b2730cf80b79bfd0e08201c33eb942f5
Reviewed-on: https://chromium-review.googlesource.com/1110450
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Andrii Shyshkalov fce7bc9e08 Add infradata/config repo to gclient.
This way it'll be discoverable by run_presubmit recipe.

R=vadimsh

Bug: 613739
Change-Id: I84ec3b84292940e74e4d707e4d257eb560004f56
Reviewed-on: https://chromium-review.googlesource.com/1102010
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov cd8d4a659a gclient recipe: use env_suffixes insetad of custom ENV manipulation.
R=iannucci@chromium.org

Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Change-Id: Ie0b042389b7e2c505cd8dbecc1a30a37244ae238
Reviewed-on: https://chromium-review.googlesource.com/1100110
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrew Grieve 1f156687a2 gerrit/api.py: Add get_revision() helper.
Allows easily getting at more than just the commit message.
I intend to look at the author to whitelist bots from binary size
checks (initially).

Bug: 702625
Change-Id: I684fdfacc88635ec4849216c74183f63f9d54810
Reviewed-on: https://chromium-review.googlesource.com/1070483
Commit-Queue: agrieve <agrieve@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lesmes 3f0591d4f1 Reland "bot_update: Patch on gclient by default."
This is a reland of e38cf570fc

Sinc the last reland catapult, custom-tab-client and syzygy had their origins updated
to the source of truth:
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1043254
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1043355

This fixes the last problem we had, where the origin in gclient was different from the
source of truth, and so the patch couldn't be applied.

Original change's description:
> bot_update: Patch on gclient by default.
>
> See [4] for the design doc for this change.
>
> We're moving the patching logic from bot_update to gclient.
> This way we can test patches of dependencies that modify their DEPS
> files as well, for example when ANGLE uploads a patch that modifies
> DEPS files to be tested by a Chromium trybot.
>
> Chromium Linux CQ and Angle trybots are already running this way
> (see [1] and [2]) and a PSA has already been sent [3]
>
> [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
>
> Bug: 643346
> Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: release_scripts
> Reviewed-on: https://chromium-review.googlesource.com/1014356
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>

Bug: 643346

Change-Id: Ia0db591c6bf9695f56e6b71777c567595986a511
Reviewed-on: https://chromium-review.googlesource.com/1031206
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Edward Lemur 3d894aa45e bot_update: Fix manifest creation.
It fails when both the revision and the url are None.

Bug: 841936
Change-Id: Idef45a015624a92226d4ccd38ed5b978bf786993
Reviewed-on: https://chromium-review.googlesource.com/1053996
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
7 years ago
Edward Lemur 31a8da265a bot_update: Assert no syncing of refs/remotes/*
Bug: 840445
Change-Id: I9c9a13914bb2da0d07ed32f71453cbb0f30b59e0
Reviewed-on: https://chromium-review.googlesource.com/1053185
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Takuto Ikuta 7c3ff1311a Introduce GIT_TRACE_PERFORMANCE
GIT_TRACE_PERFORMANCE shows invocation time of each subcommands.
https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

This is tested on
https://ci.chromium.org/swarming/task/3d605b8aab1a5a10?server=chromium-swarm.appspot.com
bot_update ran successfully.

Bug: 840592
Change-Id: I65ef733d1473f9f0f7e0109b0f727595766021b6
Reviewed-on: https://chromium-review.googlesource.com/1053028
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Takuto Ikuta 5623aeebe8 Introduce GIT_TRACE for chromium/src in bot_update step
This CL is tested on
https://ci.chromium.org/swarming/task/3d5fde091e610010?server=chromium-swarm.appspot.com

api.py change is for auto roller

Bug: 840592
Change-Id: Ib6195dc1f4539c8f02fe0696dc140b4cf0973baf
Reviewed-on: https://chromium-review.googlesource.com/1049987
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lemur 04065d39f7 Reland "bot_update: Fetch references to be synced."
This is a reland of 3806b7fbd0

Original change's description:
> bot_update: Fetch references to be synced.
>
> If a solution is to be synced to a reference, such as
> refs/branch-heads/#### fetch it so that it can later be
> checked out.
>
> I tried this change on a LUCI task that runs chromium_presubmit
> for crrev.com/c/1048868. The bot_update works as expected:
> https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com
>
>
> Bug: 840445
>
> Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/1049007
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

Bug: 840445

Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I64a97aafe5fab9a26c478262e9ca104dc59e61f5
Reviewed-on: https://chromium-review.googlesource.com/1052387
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
7 years ago
agrieve afec759dca Revert "bot_update: Fetch references to be synced."
This reverts commit 3806b7fbd0.

Reason for revert: Broke downstream tot bots

Original change's description:
> bot_update: Fetch references to be synced.
> 
> If a solution is to be synced to a reference, such as
> refs/branch-heads/#### fetch it so that it can later be
> checked out.
> 
> I tried this change on a LUCI task that runs chromium_presubmit
> for crrev.com/c/1048868. The bot_update works as expected:
> https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com
> 
> 
> Bug: 840445
> 
> Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/1049007
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

TBR=mmoss@chromium.org,machenbach@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org

Change-Id: If94df8684e634504d8a0c11b2aa5862a458ba19f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 840445, 841306
Reviewed-on: https://chromium-review.googlesource.com/1052187
Reviewed-by: agrieve <agrieve@chromium.org>
Commit-Queue: agrieve <agrieve@chromium.org>
7 years ago
Edward Lemur 3806b7fbd0 bot_update: Fetch references to be synced.
If a solution is to be synced to a reference, such as
refs/branch-heads/#### fetch it so that it can later be
checked out.

I tried this change on a LUCI task that runs chromium_presubmit
for crrev.com/c/1048868. The bot_update works as expected:
https://ci.chromium.org/swarming/task/3d5b294cff467710?server=chromium-swarm.appspot.com


Bug: 840445

Change-Id: I6444c0feb679ca89b2f4065215de915494e5082a
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/1049007
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov 5e5f2d6035 bot_update: attempt to disable GC completely on main repo.
Motivation: see https://crbug.com/822437#c6
tl;dr apparently, auotpacking still happens, this CL tries to stop
prevent it harder than before.

R=vadimsh@chromium.org

Bug: 822437
Change-Id: Idda384bdebd96c10ac79c48b0622f84550060362
Reviewed-on: https://chromium-review.googlesource.com/1045491
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Edward Lemur ea240053f9 gclient: Update catapult and custom-tab-client origin URLs
Otherwise, the origin URLs for the checkout and the patch won't match
and applying the patch on gclient will fail.

Bug: 643346
Change-Id: If005bb10e579b3d50f9b6d0ff39440bb9a300cbd
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/1043254
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Stephen Martinis ea50265aef Revert "Reland "bot_update: Patch on gclient by default.""
This reverts commit 8f20428bf4.

Reason for revert: Probably broke bot update on catapult bots: https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Linux%20Tryserver/builds/11225

Original change's description:
> Reland "bot_update: Patch on gclient by default."
> 
> This is a reland of e38cf570fc
> 
> The issue with win CQ bots was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1036309
> It was due to misconfigured git user.{name,email}
> 
> Original change's description:
> > bot_update: Patch on gclient by default.
> >
> > See [4] for the design doc for this change.
> >
> > We're moving the patching logic from bot_update to gclient.
> > This way we can test patches of dependencies that modify their DEPS
> > files as well, for example when ANGLE uploads a patch that modifies
> > DEPS files to be tested by a Chromium trybot.
> >
> > Chromium Linux CQ and Angle trybots are already running this way
> > (see [1] and [2]) and a PSA has already been sent [3]
> >
> > [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> > [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> > [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> > [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
> >
> > Bug: 643346
> > Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> > Recipe-Nontrivial-Roll: skia
> > Recipe-Nontrivial-Roll: infra
> > Recipe-Nontrivial-Roll: build_limited_scripts_slave
> > Recipe-Nontrivial-Roll: skiabuildbot
> > Recipe-Nontrivial-Roll: build
> > Recipe-Nontrivial-Roll: release_scripts
> > Reviewed-on: https://chromium-review.googlesource.com/1014356
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> > Reviewed-by: Aaron Gable <agable@chromium.org>
> 
> Bug: 643346
> Change-Id: I4f4edd8e815cb12a4cbc3ab690eaf588bc2accc1
> Reviewed-on: https://chromium-review.googlesource.com/1039966
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=agable@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org

Change-Id: Ia8a5779cde3c31812121cffdd806476d4536f35e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 643346
Reviewed-on: https://chromium-review.googlesource.com/1043167
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
7 years ago
Edward Lesmes 8f20428bf4 Reland "bot_update: Patch on gclient by default."
This is a reland of e38cf570fc

The issue with win CQ bots was fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1036309
It was due to misconfigured git user.{name,email}

Original change's description:
> bot_update: Patch on gclient by default.
>
> See [4] for the design doc for this change.
>
> We're moving the patching logic from bot_update to gclient.
> This way we can test patches of dependencies that modify their DEPS
> files as well, for example when ANGLE uploads a patch that modifies
> DEPS files to be tested by a Chromium trybot.
>
> Chromium Linux CQ and Angle trybots are already running this way
> (see [1] and [2]) and a PSA has already been sent [3]
>
> [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
>
> Bug: 643346
> Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: release_scripts
> Reviewed-on: https://chromium-review.googlesource.com/1014356
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>

Bug: 643346
Change-Id: I4f4edd8e815cb12a4cbc3ab690eaf588bc2accc1
Reviewed-on: https://chromium-review.googlesource.com/1039966
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur 6c692edf01 Reland "bot_update: Set user.{name,email} when applying patch refs."
This is a reland of e86fe982e8

Original change's description:
> bot_update: Set user.{name,email} when applying patch refs.
> 
> git rebase complains when user name/email has not been configured, as was
> the case for the Chromium trybot.
> 
> Bug: 643346
> Change-Id: Iab7dac0a98608e09b63a53d5bacb98925701fbfd
> Reviewed-on: https://chromium-review.googlesource.com/1031524
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Bug: 643346
Change-Id: I4b4d07675222d39d704560ed84f9c221e87aba8b
Reviewed-on: https://chromium-review.googlesource.com/1036309
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Michael Moss 1eb58e1f88 Add a recipe config to enable the bot_update "--with_tags" flag.
This is needed for release builders to checkout the release tags that
will be created when the buildspec branch/release info is migrated out
to the regular repos.

BUG=825063
R=jbudorick@google.com

Change-Id: I5f0e4cdd94de11c59f9b4a5c11aabea129e81523
Reviewed-on: https://chromium-review.googlesource.com/1036526
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
7 years ago
Stephen Martinis 43d72759fe Revert "bot_update: Set user.{name,email} when applying patch refs."
This reverts commit e86fe982e8.

Reason for revert: Probably causing outages on win7_chromium_rel_ng.

Original change's description:
> bot_update: Set user.{name,email} when applying patch refs.
> 
> git rebase complains when user name/email has not been configured, as was
> the case for the Chromium trybot.
> 
> Bug: 643346
> Change-Id: Iab7dac0a98608e09b63a53d5bacb98925701fbfd
> Reviewed-on: https://chromium-review.googlesource.com/1031524
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=agable@chromium.org,ehmaldonado@chromium.org

Change-Id: Ic57398323476fccef34f766270d760c3975630a6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 643346
Reviewed-on: https://chromium-review.googlesource.com/1036308
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
7 years ago
Edward Lesmes e86fe982e8 bot_update: Set user.{name,email} when applying patch refs.
git rebase complains when user name/email has not been configured, as was
the case for the Chromium trybot.

Bug: 643346
Change-Id: Iab7dac0a98608e09b63a53d5bacb98925701fbfd
Reviewed-on: https://chromium-review.googlesource.com/1031524
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Andrii Shyshkalov c6e57bb748 bot_update: really check whether repo is pinned at an actual sha1 revision.
Previously, anything not equal to HEAD meant pinned revision.
This backfired on values like "origin/master" which are meant as
equivalent to HEAD, but which bot_update tried to ensure exist in
checkout, which was refreshed several times under assumption that
this is eventual consistency of git servers.

Instead, this CL treats repo as pinned if and only if the treeish
of the repo is matching sha1 regex.

R=hinoka@chromium.org

Bug: 838292
Change-Id: I9cb46535f46b2898cf54900ede7960ba08770d77
Post-Scriptum: git bisect + Led is awesome, took only 8 manual steps.
Reviewed-on: https://chromium-review.googlesource.com/1035793
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Ben Pastene 7580f9dac9 Revert "Reland "bot_update: Patch on gclient by default.""
This reverts commit e38cf570fc.

Reason for revert: likely broke bot_update on win CQ bots:
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/154654
https://bugs.chromium.org/p/chromium/issues/detail?id=837321

Original change's description:
> Reland "bot_update: Patch on gclient by default."
> 
> This is a reland of 355b11a31f
> 
> Original change's description:
> > bot_update: Patch on gclient by default.
> > 
> > See [4] for the design doc for this change.
> > 
> > We're moving the patching logic from bot_update to gclient.
> > This way we can test patches of dependencies that modify their DEPS
> > files as well, for example when ANGLE uploads a patch that modifies
> > DEPS files to be tested by a Chromium trybot.
> > 
> > Chromium Linux CQ and Angle trybots are already running this way
> > (see [1] and [2]) and a PSA has already been sent [3]
> > 
> > [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> > [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> > [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> > [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
> > 
> > Bug: 643346
> > Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> > Recipe-Nontrivial-Roll: skia
> > Recipe-Nontrivial-Roll: infra
> > Recipe-Nontrivial-Roll: build_limited_scripts_slave
> > Recipe-Nontrivial-Roll: skiabuildbot
> > Recipe-Nontrivial-Roll: build
> > Recipe-Nontrivial-Roll: release_scripts
> > Reviewed-on: https://chromium-review.googlesource.com/1014356
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> > Reviewed-by: Aaron Gable <agable@chromium.org>
> 
> Bug: 643346
> Change-Id: I2da17456f24d1e9fcc3f48f9242b4db05cadbf6d
> Reviewed-on: https://chromium-review.googlesource.com/1029073
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=agable@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org

Change-Id: I33e4bf7b5f416f6356dbd0b50a276cd36dd4e9b3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 643346
Reviewed-on: https://chromium-review.googlesource.com/1030718
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
7 years ago
Edward Lesmes e38cf570fc Reland "bot_update: Patch on gclient by default."
This is a reland of 355b11a31f

Original change's description:
> bot_update: Patch on gclient by default.
> 
> See [4] for the design doc for this change.
> 
> We're moving the patching logic from bot_update to gclient.
> This way we can test patches of dependencies that modify their DEPS
> files as well, for example when ANGLE uploads a patch that modifies
> DEPS files to be tested by a Chromium trybot.
> 
> Chromium Linux CQ and Angle trybots are already running this way
> (see [1] and [2]) and a PSA has already been sent [3]
> 
> [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
> 
> Bug: 643346
> Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: release_scripts
> Reviewed-on: https://chromium-review.googlesource.com/1014356
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>

Bug: 643346
Change-Id: I2da17456f24d1e9fcc3f48f9242b4db05cadbf6d
Reviewed-on: https://chromium-review.googlesource.com/1029073
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Andrii Shyshkalov 3cb0767ac1 recipes: cache destination branch of a Gerrit change.
It doesn't change after change is created.

R=jchinlee@chromium.org

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I4c4758085f6cfa7c2ef5267c6c71c9b324263f95
Reviewed-on: https://chromium-review.googlesource.com/1029227
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
7 years ago
Edward Lesmes 2acf495798 Revert "bot_update: Patch on gclient by default."
This reverts commit 355b11a31f.

Reason for revert:
Easier to do this another way that doesn't cause expectation changes,
so it doesn't require non-trivial rolls.

Original change's description:
> bot_update: Patch on gclient by default.
> 
> See [4] for the design doc for this change.
> 
> We're moving the patching logic from bot_update to gclient.
> This way we can test patches of dependencies that modify their DEPS
> files as well, for example when ANGLE uploads a patch that modifies
> DEPS files to be tested by a Chromium trybot.
> 
> Chromium Linux CQ and Angle trybots are already running this way
> (see [1] and [2]) and a PSA has already been sent [3]
> 
> [1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
> [2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
> [3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
> [4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#
> 
> Bug: 643346
> Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: release_scripts
> Reviewed-on: https://chromium-review.googlesource.com/1014356
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>

TBR=agable@chromium.org,ehmaldonado@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 643346
Change-Id: I0e694b0e398770b8bd2e5a3170e158a7e54c5a61
Reviewed-on: https://chromium-review.googlesource.com/1029310
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lesmes 355b11a31f bot_update: Patch on gclient by default.
See [4] for the design doc for this change.

We're moving the patching logic from bot_update to gclient.
This way we can test patches of dependencies that modify their DEPS
files as well, for example when ANGLE uploads a patch that modifies
DEPS files to be tested by a Chromium trybot.

Chromium Linux CQ and Angle trybots are already running this way
(see [1] and [2]) and a PSA has already been sent [3]

[1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#214
[2] https://chromium.googlesource.com/angle/angle/+/infra/config/cr-buildbucket.cfg#105
[3] https://groups.google.com/a/google.com/forum/#!topic/chrome-infra/FVHN2kdUX4I
[4] https://docs.google.com/document/d/1GSTw4ysm5AYjNR6vMA7NJUxLK2VN9IpOEp_v0u-wyOE/edit?ts=5ad78165#

Bug: 643346
Change-Id: Ic9a9dcfae86aca730d35fbb26c26b21135e05bf3
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: release_scripts
Reviewed-on: https://chromium-review.googlesource.com/1014356
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
John Budorick 9ccad5689c bot_update: Ensure create_manifest ignores repos with explicit None urls.
Bug: 830306
Change-Id: Id87a6f2053c7e3cd489647d1cbbc8a83f2e585db
Reviewed-on: https://chromium-review.googlesource.com/1003326
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Edward Lesmes d69c5dd538 bot_update: Don't patch if gerrit_repo or gerrit_ref are empty/None.
We should apply the patch only when gerrit_repo and gerrit_ref are passed,
instead of using only the --apply-patch-on-gclient flag.

R=agable@chromium.org

Bug: 643346
Change-Id: I28547aed9daeefa2657a108e0bacbfd4fc9fa07e
Reviewed-on: https://chromium-review.googlesource.com/989667
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Nico Weber 6ceb345389 Remove dnj from depot_tools OWNERS
Add some other people to recipes/OWNERS

Change-Id: I554d1fd354e628d2e62629be7ce20e899364c4a9
Reviewed-on: https://chromium-review.googlesource.com/984972
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Nico Weber 7bfbb43277 add a standalone angle gclient config
To be used in https://chromium-review.googlesource.com/c/chromium/tools/build/+/984433

Bug: 820421
Change-Id: I76d32ed9f7b5caa256bfa85220ac0c256bbc1a48
Reviewed-on: https://chromium-review.googlesource.com/984692
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Edward Lesmes ee04b9a2f4 bot_update: Make gclient experiment a property instead of an argument.
Is easier to configure experimental bots this way.

Bug: 643346
Change-Id: Idb6d3a68b02949dce71dbcba38e8ef756c467830
Reviewed-on: https://chromium-review.googlesource.com/981515
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Edward Lesmes 385a4fdd46 bot_update: Add a flag to move applying the patch to gclient.
This adds a --enable-gclient-experiment flags that tells bot_update.py
to skip applying the patch, and instead forward the flags to gclient.

Bug: 643346
Change-Id: Ia4275a126e6adba54dfcc894d224c50c166db90e
Reviewed-on: https://chromium-review.googlesource.com/962938
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Andrii Shyshkalov b3065fb6a1 bot_update: don't repack, don't gc in main repo of builder's checkout.
The main purpose is to avoid running slow and useless operation in
chromium/src checkouts. Ideally, this should be done in every DEPS-ed
repo, too, but that requires substantially more changes in gclient.
Thus, this CL provides biggest bang for the buck.

Example led build: https://ci.chromium.org/swarming/task/3c62efb31ed3a910?server=chromium-swarm.appspot.com

R=hinoka@chromium.org

Bug: 822437
Change-Id: I347d85e5cb5158b77969a755b67ca2289fb39d40
Reviewed-on: https://chromium-review.googlesource.com/973900
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
7 years ago
Robert Iannucci 456b0d648f [bot_update] Remove support for rietveld patches.
R=agable@chromium.org, tandrii@chromium.org

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Change-Id: I518335e7b8fdce316b5566caccd3c4c7c2ae741a
Reviewed-on: https://chromium-review.googlesource.com/960161
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Robert Iannucci f4c2703a6d [bot_update] Remove rietveld recipe_module.
R=agable@chromium.org, tandrii@chromium.org

Recipe-Nontrivial-Roll: build
Change-Id: Ic4c2dc7a50d389f18f3e0c1de332cb1a0ab376a9
Reviewed-on: https://chromium-review.googlesource.com/957834
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Robert Iannucci 3a8b09088b [bot_update] Remove use_site_config_creds.
This method hasn't been supported for a long time now (and besides, its
only for Rietveld).

R=agable@chromium.org, tandrii@chromium.org

Recipe-Manual-Change: infra
Change-Id: Ie6e63834dca67962db29f2cb407950ed85db55a7
Reviewed-on: https://chromium-review.googlesource.com/957832
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Michael Achenbach 9b4d0c7715 Gerrit: Enable overriding step_test_data
Bug: v8:7419
Change-Id: I49ad9358b9487b76503c3c1f4d6838aa0dd42736
Reviewed-on: https://chromium-review.googlesource.com/938969
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
7 years ago
Tom Anderson 592962c7af Write target_cpu to .gclient
Also, some lines went over 80chars, so I ran 'yapf --style chromium' which
formatted some unrelated things.

BUG=905285
R=dpranke

Change-Id: Iee5f46d88a6e9782612cc4f9e5a2cb72d62ab6af
Reviewed-on: https://chromium-review.googlesource.com/907736
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
7 years ago
Tom Anderson 2dd5473557 Add target_cpu and target_cpu_only to gclient base config
BUG=807986
R=dpranke

Change-Id: I6b7e94d489a228c5dde23a9b6c7d6f58ce73b180
Reviewed-on: https://chromium-review.googlesource.com/905285
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
7 years ago
Andrii Shyshkalov 5eab58ee90 Revert "bot_update: allow rebasing the patch onto an older revision."
This reverts commit fa573785df.

Reason for revert: broke CQ on branches.
Example: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/28220

Original change's description:
> bot_update: allow rebasing the patch onto an older revision.
>
> When applying a rebase, we normally go from an older base commit to a newer one.
>
> A ---- B ---- C ---- D -origin/master
>                \
>                 E -branch
>
> In this case, `git rebase D` would certainly work as expected. However,
> writing `git rebase B` would NOT get us to the following state:
>
> A ---- B ---- C ---- D -origin/master
>         \
>          E' -branch
>
> In fact, it would have no effect.
>
> This article http://matthew-brett.github.io/pydagogue/rebase_without_tears.html
> explains the general invocation as
> > `git rebase --onto <graft-point> <exclude-from> <include-from>`
> > If you don’t specify --onto, <graft-point> defaults to <exclude-from>
>
> So what's happening is, by writing `git rebase B` we're rebasing onto B,
> excluding commits that are in B. Commit C is not "in" B so it is kept and we're
> back to the starting point.
>
> So I suggest to change the invocation to `git rebase --onto B origin/master`,
> which rebases onto B, excluding commits that are in origin/master. This works
> more generally and allows rebasing "backwards".
>
> Bug: None
> Change-Id: I68e4d805811530b585550bc75099354fef4e9c15
> Reviewed-on: https://chromium-review.googlesource.com/904004
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Oleh Prypin <oprypin@chromium.org>

TBR=iannucci@chromium.org,tandrii@chromium.org,oprypin@chromium.org

Change-Id: I7243641d84428bf4504ee798bf31ad0afbfd4865
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: None
Reviewed-on: https://chromium-review.googlesource.com/905862
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Andrii Shyshkalov cd33775a7a bot_update: optimize checkout of solution's main repo.
Goal: reduce number of git fetches from remote to the minimum possible.
Minimum possible is either:
 * 0 if desired revision is pinned (!= HEAD) and already exists in cache
 * 1 if desired revision is HEAD
 * >= 2 if revision is pinned but we talk temporarily
   out-of-date git server not yet having it.

This CL achieves the above.

No fetch:
  https://ci.chromium.org/swarming/task/39dad20230c84110?server=chromium-swarm.appspot.com
  bot_update takes <1min, but it used to take at least 2. The steps are now like this:

  ...
  ===Running git cat-file -e 33bf8a94dcd5e0abbdf83e4afaed24b5180e3eb2===
  In directory: /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
  ===Succeeded in 0.0 mins===
  ===Running /b/swarming/w/ir/cipd_bin_packages/bin/python -u /b/swarming/w/ir/recipe-checkout-dir/depot_tools/git_cache.py exists --quiet --cache-dir /b/swarming/w/ir/cache/git https://chromium.googlesource.com/chromium/src.git===
  In directory: /b/swarming/w/ir/cache/builder/linux
  /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
  ===Succeeded in 0.0 mins===
  ===Running git remote set-url origin /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src===
  In directory: /b/swarming/w/ir/cache/builder/linux/src
  ===Succeeded in 0.0 mins===
  ===Running git fetch origin===
  In directory: /b/swarming/w/ir/cache/builder/linux/src
  From /b/swarming/w/ir/cache/git/chromium.googlesource.com-chromium-src
     003eedbfee47..5210107a20e8  master     -> origin/master
  ===Succeeded in 0.0 mins===
  ===Running git checkout --force 33bf8a94dcd5e0abbdf83e4afaed24b5180e3eb2===
  ...

Fetch & success:
  https://ci.chromium.org/swarming/task/39db022b874ffd10?server=chromium-swarm.appspot.com

Fetch & retry because missing rev:
  https://ci.chromium.org/swarming/task/39db04ce8b4b8910?server=chromium-swarm.appspot.com

Fetch because rev=HEAD:
  https://ci.chromium.org/swarming/task/39db0c0dd5ab5d10?server=chromium-swarm.appspot.com

This has already been adopted in gclient.

Bug:
Change-Id: Id99892b62719fdf3f7e6e59058986d1500384f8d
Reviewed-on: https://chromium-review.googlesource.com/771591
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
7 years ago
Oleh Prypin fa573785df bot_update: allow rebasing the patch onto an older revision.
When applying a rebase, we normally go from an older base commit to a newer one.

A ---- B ---- C ---- D -origin/master
               \
                E -branch

In this case, `git rebase D` would certainly work as expected. However,
writing `git rebase B` would NOT get us to the following state:

A ---- B ---- C ---- D -origin/master
        \
         E' -branch

In fact, it would have no effect.

This article http://matthew-brett.github.io/pydagogue/rebase_without_tears.html
explains the general invocation as
> `git rebase --onto <graft-point> <exclude-from> <include-from>`
> If you don’t specify --onto, <graft-point> defaults to <exclude-from>

So what's happening is, by writing `git rebase B` we're rebasing onto B,
excluding commits that are in B. Commit C is not "in" B so it is kept and we're
back to the starting point.

So I suggest to change the invocation to `git rebase --onto B origin/master`,
which rebases onto B, excluding commits that are in origin/master. This works
more generally and allows rebasing "backwards".

Bug: None
Change-Id: I68e4d805811530b585550bc75099354fef4e9c15
Reviewed-on: https://chromium-review.googlesource.com/904004
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Oleh Prypin <oprypin@chromium.org>
7 years ago
Lei Zhang f2b5774fef Add got_revision key to pdfium buildbot config.
This is needed for run_presubmit.py to work.

BUG=pdfium:971

Change-Id: Ib598fbfd6eab3c0b7628301eb3bd54c548a4885e
Reviewed-on: https://chromium-review.googlesource.com/898039
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>
7 years ago
Michael Moss b13fba7efb Allow bools in gclient recipe custom_vars.
This has been supported in gclient, and used in DEPS, since
https://crrev.com/c/687496.

R=dpranke@google.com, iannucci@google.com

Bug: 570091,807318
Change-Id: I59935625e3bae6010a371ad7bb1413e177a09176
Reviewed-on: https://chromium-review.googlesource.com/896835
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
7 years ago
Takuto Ikuta 4fbf4bece2 Remove windows specific git pack limit
This is full version of
https://chromium-review.googlesource.com/823544

After the CL landing, we see no performance regression in bot_update step.
And regular 95%ile in 1h window improved from more than 300s to less than or around 200s.
See tandrii's query in https://bugs.chromium.org/p/chromium/issues/detail?id=749709#c91 for detail.

Let me reduce frequency of slow bot_update step more like below.
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/69844
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/69886
https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/69892

Bug: 749709
Change-Id: I44474aa9ecdd733d68fd13fe8b791820fb80381f
Reviewed-on: https://chromium-review.googlesource.com/838766
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Aaron Gable de8ce79f79 Remove rietveld support from tryserver recipe module
Bug: 770408
Recipe-Nontrivial-Roll: build
Change-Id: Icafd4bd11009975c40e53d93901589c07ecd738e
Reviewed-on: https://chromium-review.googlesource.com/707723
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
7 years ago
recipe-roller ac4e7911c0 Roll recipe dependencies (nontrivial).
This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).


Please review the expectation changes, and LGTM+CQ.


More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
recipe_engine:
  https://crrev.com/77667b5ba38c4d205a743d04eb625ac068bdd78e Always use Gerrit for tryserver recipe tests (agable@chromium.org)


R=dnj@chromium.org, martiniss@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I2df24323974ae0d3b3b0524240b64be5740977c6
Reviewed-on: https://chromium-review.googlesource.com/836760
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Aaron Gable aac382b3b6 Remove rietveld-based tests from tryserver recipe module
This introduces a pragma: no cover, which is unfortunate.
But this is a prerequisite for removing rietveld support
entirely from the recipe engine without having to do
manual recipe rolls. The entire block of uncovered code
will be removed shortly.

Bug: 770408
Change-Id: I37319f3ae70413adcda655c443325bccffe695d8
Reviewed-on: https://chromium-review.googlesource.com/834455
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
7 years ago
Takuto Ikuta 9fce213bdb Partial revert of "git_cache: lower max num of .pack files before re-bootstrap on Win."
This partially reverts commit d51ed57edb.

Reason for revert:
New git client for windows was rolled including fix for slow `git fetch`.
I guess smaller pack limit causes frequent bootstrap taking 2~3 minutes longer than
the case it does not happen.
Let me see what happen if we increase pack limit 9 -> 30.
I will increase this to 50 if this won't cause regression again.

Original change's description:
> git_cache: lower max num of .pack files before re-bootstrap on Win.
>
> It used to be 50, I think ~9 gives best results for Chromium on Win:
>   on golo VM, it takes <4 minutes to re-boostrap + git fetch small
>   delta, assuming zipped git checkout for bootstrap is fresh (~1day).
>
> For other repos, which are significantly smaller, this change should
> have minor effect if at all.
>
> Test: I tested this using `led` tool on Win7 machines running LUCI
> stack extensively. For example,
>
>   * https://ci.chromium.org/swarming/task/3a0102e8c8657410
>     shows case with few .pack files, hence just 1 fetch
>
>   * https://ci.chromium.org/swarming/task/3a010282f9fd8010
>     shows case with 39 .pack files and so bootstrapping + fetch.
>     If you look at prior tasks on the same VM, you'd find this:
>     https://ci.chromium.org/swarming/task/39ffe843d01ed010
>     which spent 8 minutes doing 1 incremental fetch with 39 .pack
>     files.
>
> **Troopers/Sheriffs**: This change is safe to revert.
> However, beware that you should also at the same time revert the recipe
> roll of this CL to the repo, in which the failed builder's recipe is
> located, typically `chromium/tools/build`.
>
> Bug: 749709
> Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
> Reviewed-on: https://chromium-review.googlesource.com/787174
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 749709
Change-Id: I3052abe4a9b53277a60c0791a85355e7a0bbdf8f
Reviewed-on: https://chromium-review.googlesource.com/823544
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
7 years ago
Robert Iannucci 41d9d87e96 [recipe_modules/depot_tools] Add on_path helper.
This will be used by various things in build and build_internal which
"need" a depot_tools on PATH (though I suspect most of those should be
refactored to use the depot_tools in chromium/src.git).

R=tandrii@chromium.org

Bug: 789808
Change-Id: I5d0590ccdc5482bd8e3048ad46a39b5aedacbddb
Reviewed-on: https://chromium-review.googlesource.com/828465
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
7 years ago
Robert Iannucci 069dd4fa6b [bot_update] Put depot_tools at the end of $PATH on LUCI.
This should prevent bot_update from overriding vpython by accident.

R=tandrii@chromium.org

Recipe-Nontrivial-Roll: build
Bug: 789808
Change-Id: Ia574204439588862f29f94a80e978b2ff4b57586
Reviewed-on: https://chromium-review.googlesource.com/828069
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Aaron Gable 7817f02543 Reland "Use core.quotePath=false when git is listing files"
This is a reland of 9219d35688

The original was reverted due to a typo (core,quotePath instead
of core.quotePath). This version is fixed.

Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
>
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I28d2260948aaf63bd865888c2f60e4cdee9aea48
Reviewed-on: https://chromium-review.googlesource.com/822990
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
7 years ago
Robbie Iannucci 6aaee3065c Revert "Use core.quotePath=false when git is listing files"
This reverts commit 9219d35688.

Reason for revert: unfortunately this says "core,quotePath" and since it includes recipe changes, we need something that the roller can munch on :(

Original change's description:
> Use core.quotePath=false when git is listing files
> 
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
> 
> R=​maruel
> 
> Bug: 792302
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

TBR=maruel@chromium.org,agable@chromium.org

Change-Id: I226388f19024403240a1443eb2b878b9293220e1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 792302
Reviewed-on: https://chromium-review.googlesource.com/821671
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
8 years ago
Aaron Gable 9219d35688 Use core.quotePath=false when git is listing files
This prevents git from putting quotes around some file names
(those that have astral-plane characters) and not around others.

R=maruel

Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
Reviewed-on: https://chromium-review.googlesource.com/815454
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
8 years ago
Andrii Shyshkalov 009150b7af bot_update: be explicit when checking out branches.
$ git checkout <name>

may be interpreted by git as checking out a file named <name>
if <name> has slashes. Thus,

$ git checkout infra/config

ends up checking out infra/config directory of the (typically)
master branch that was checked out before. To avoid confusion,
we have to add '--' separator between branch name and file names:

$ git checkout <name> --

This in turns allows to run CQ-based presubmit on infra/config branches.

R=nodir@chromium.org

Bug: 790738
Change-Id: I6ce31a8f0fbd66fd59ac7c2ea9cccd3ff97d1f0e
Reviewed-on: https://chromium-review.googlesource.com/802136
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
8 years ago
Andrii Shyshkalov 1b30125fbc bot_update: don't remove git cache's .lock files.
bot_update and gclient invokes git cache with --ignore-locks.
This should also reduce noisiness of bot_update stdout.

R=iannucci@chromium.org

Bug:
Change-Id: I6baf9e5665d244d08aab58a7d3327e6c3204b2b5
Reviewed-on: https://chromium-review.googlesource.com/791536
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 46a672bafa gclient: don't update cache if needed revision is already there.
Tested end-to-end, for example
  https://ci.chromium.org/swarming/task/3a0147207378b910
which contains:
  src/media/cdm/api (Elapsed: 0:00:01)
  ----------------------------------------
  [0:00:00] Started.
  _____ src\media\cdm\api at ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237
  [0:00:00] running "git cat-file -e ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237" in "e:\b\s\w\ir\cache\git\chromium.googlesource.com-chromium-cdm"
  skipping mirror update, it has rev=ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237 already

thereby saving on needless git fetch (~40s in glcient sync on win trybots),
and reducing the rate of .pack file accumulation inside cache directories.


Risks: silently broken recipes which run gclient sync (or worse, bot_update)
as a means of fetching latest commits in all repos of a solution. I think
the benefit of faster bot_update in chromium CQ is worth the potential risk.

PSA: https://groups.google.com/a/chromium.org/d/msg/infra-dev/UYLdBwAXm1Y/OV9QB6JnBQAJ

Bug: 749709
Change-Id: I7a9e8ab82a5e2b848e450f19a798ac18a0b5e201
Reviewed-on: https://chromium-review.googlesource.com/787331
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
8 years ago
Andrii Shyshkalov 568f473e4b bot_update: add mooooooore chars in logs.
Before, bot_update would print after command:
  Succeeded (Failed) in 0.1 mins
but figuring out which command takes a lot of scrolling/searching
for previous
  Running <cmd with args>
This CL fixes that:
  Succeeded (Failed) in 0.1 mins <cmd with args>



R=machenbach@chromium.org, maruel@chromium.org
TEST=end-to-end with led tool, but only on LUCI stack.

Bug: 749709
Change-Id: Id3031a12b9013ab136ea03cdf6d325930722e50f
Reviewed-on: https://chromium-review.googlesource.com/787175
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov d51ed57edb git_cache: lower max num of .pack files before re-bootstrap on Win.
It used to be 50, I think ~9 gives best results for Chromium on Win:
  on golo VM, it takes <4 minutes to re-boostrap + git fetch small
  delta, assuming zipped git checkout for bootstrap is fresh (~1day).

For other repos, which are significantly smaller, this change should
have minor effect if at all.

Test: I tested this using `led` tool on Win7 machines running LUCI
stack extensively. For example,

  * https://ci.chromium.org/swarming/task/3a0102e8c8657410
    shows case with few .pack files, hence just 1 fetch

  * https://ci.chromium.org/swarming/task/3a010282f9fd8010
    shows case with 39 .pack files and so bootstrapping + fetch.
    If you look at prior tasks on the same VM, you'd find this:
    https://ci.chromium.org/swarming/task/39ffe843d01ed010
    which spent 8 minutes doing 1 incremental fetch with 39 .pack
    files.

**Troopers/Sheriffs**: This change is safe to revert.
However, beware that you should also at the same time revert the recipe
roll of this CL to the repo, in which the failed builder's recipe is
located, typically `chromium/tools/build`.

Bug: 749709
Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
Reviewed-on: https://chromium-review.googlesource.com/787174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
8 years ago
Andrii Shyshkalov 0083130c97 Whitespace change inside recipes to kick recipe autoroller.
TBR=iannucci@chromium.org

Bug: 749709
Change-Id: I8aa02f780c5f2a3c2f8c0c408f83b061ba11c672
Reviewed-on: https://chromium-review.googlesource.com/788244
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 25b1da95b3 Revert "Show GIT_TRACE for git command in bot_update.py"
This reverts commit 6e215cefe2.

Reason for revert: seems to have broken bot_update 
https://ci.chromium.org/swarming/task/39ff1814b3bb5b10?server=chromium-swarm.appspot.com

Original change's description:
> Show GIT_TRACE for git command in bot_update.py
> 
> GIT_TRACE=true tells us which internal git command is slow.
> https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables
> 
> I got following results in my local repository.
> Let me confirm the bots show similar behavior.
> 
> C:\src\chromium\src>set GIT_TRACE=true
> 
> C:\src\chromium\src>git rebase-update
> 21:32:36.519698 git.c:576               trace: exec: 'git-rebase-update'
> 21:32:36.519698 run-command.c:626       trace: run_command: 'git-rebase-update'
> 21:32:38.889848 git.c:344               trace: built-in: git 'fetch' '--multiple' 'origin'
> 21:32:38.893846 run-command.c:626       trace: run_command: 'fetch' '--append' 'origin'
> Fetching origin
> 21:32:38.911858 git.c:344               trace: built-in: git 'fetch' '--append' 'origin'
> 21:32:38.983909 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:32:39.002923 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:32:39.003923 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:32:43.782266 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> 21:32:43.827301 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
> remote: Counting objects: 74400, done
> remote: Finding sources: 100% (19/19)
> 21:35:49.348951 run-command.c:626       trace: run_command: 'unpack-objects' '--pack_header=2,19'
> 21:35:49.368985 git.c:344               trace: built-in: git 'unpack-objects' '--pack_header=2,19'
> remote: Total 19 (delta 1), reused 18 (delta 1)
> Unpacking objects: 100% (19/19), done.
> 21:35:49.597328 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> 21:35:49.612375 git.c:344               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> From https://chromium.googlesource.com/chromium/src
>    e8994685157b..fd56d3ad0528  master     -> origin/master
> 21:35:50.281845 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
> 21:35:50.287848 run-command.c:1484      run_processes_parallel: done
> 21:35:50.309865 run-command.c:626       trace: run_command: 'gc' '--auto'
> 21:35:50.328880 git.c:344               trace: built-in: git 'gc' '--auto'
> 21:35:50.453967 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
> 21:35:50.460971 run-command.c:1484      run_processes_parallel: done
> 21:35:50.460971 run-command.c:626       trace: run_command: 'gc' '--auto'
> 21:35:50.480985 git.c:344               trace: built-in: git 'gc' '--auto'
> Rebasing: direct_goma
> Rebasing: python_pool
> 
> C:\src\chromium\src>git fetch
> 21:36:23.409703 git.c:344               trace: built-in: git 'fetch'
> 21:36:23.486758 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:36:23.506772 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:36:23.506772 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
> 21:36:28.542596 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> 21:36:28.578620 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
> 21:36:28.601635 git.c:344               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
> remote: Counting objects: 67598, done
> remote: Finding sources: 100% (28/28)
> 21:39:29.787441 run-command.c:626       trace: run_command: 'unpack-objects' '--pack_header=2,28'
> 21:39:29.812462 git.c:344               trace: built-in: git 'unpack-objects' '--pack_header=2,28'
> remote: Total 28 (delta 2), reused 20 (delta 2)
> Unpacking objects: 100% (28/28), done.
> 21:39:30.083690 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> 21:39:30.108704 git.c:344               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
> From https://chromium.googlesource.com/chromium/src
>    fd56d3ad0528..fbcc09282dc8  master     -> origin/master
> 21:39:30.788186 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
> 21:39:30.795191 run-command.c:1484      run_processes_parallel: done
> 21:39:30.813200 run-command.c:626       trace: run_command: 'gc' '--auto'
> 21:39:30.836212 git.c:344               trace: built-in: git 'gc' '--auto'
> 
> 
> Bug:749709
> Change-Id: I5b94549c42271885365c2354dc3f58399d6aebe0
> Reviewed-on: https://chromium-review.googlesource.com/786691
> Reviewed-by: Ryan Tseng <hinoka@chromium.org>
> Commit-Queue: Takuto Ikuta <tikuta@google.com>

TBR=hinoka@chromium.org,agable@chromium.org,tandrii@chromium.org,tikuta@google.com

Change-Id: I970ddac1fd21e904c69dc18e48169d49293b1800
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 749709
Reviewed-on: https://chromium-review.googlesource.com/786413
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Takuto Ikuta 6e215cefe2 Show GIT_TRACE for git command in bot_update.py
GIT_TRACE=true tells us which internal git command is slow.
https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

I got following results in my local repository.
Let me confirm the bots show similar behavior.

C:\src\chromium\src>set GIT_TRACE=true

C:\src\chromium\src>git rebase-update
21:32:36.519698 git.c:576               trace: exec: 'git-rebase-update'
21:32:36.519698 run-command.c:626       trace: run_command: 'git-rebase-update'
21:32:38.889848 git.c:344               trace: built-in: git 'fetch' '--multiple' 'origin'
21:32:38.893846 run-command.c:626       trace: run_command: 'fetch' '--append' 'origin'
Fetching origin
21:32:38.911858 git.c:344               trace: built-in: git 'fetch' '--append' 'origin'
21:32:38.983909 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:32:39.002923 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:32:39.003923 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:32:43.782266 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
21:32:43.827301 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
remote: Counting objects: 74400, done
remote: Finding sources: 100% (19/19)
21:35:49.348951 run-command.c:626       trace: run_command: 'unpack-objects' '--pack_header=2,19'
21:35:49.368985 git.c:344               trace: built-in: git 'unpack-objects' '--pack_header=2,19'
remote: Total 19 (delta 1), reused 18 (delta 1)
Unpacking objects: 100% (19/19), done.
21:35:49.597328 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
21:35:49.612375 git.c:344               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
From https://chromium.googlesource.com/chromium/src
   e8994685157b..fd56d3ad0528  master     -> origin/master
21:35:50.281845 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
21:35:50.287848 run-command.c:1484      run_processes_parallel: done
21:35:50.309865 run-command.c:626       trace: run_command: 'gc' '--auto'
21:35:50.328880 git.c:344               trace: built-in: git 'gc' '--auto'
21:35:50.453967 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
21:35:50.460971 run-command.c:1484      run_processes_parallel: done
21:35:50.460971 run-command.c:626       trace: run_command: 'gc' '--auto'
21:35:50.480985 git.c:344               trace: built-in: git 'gc' '--auto'
Rebasing: direct_goma
Rebasing: python_pool

C:\src\chromium\src>git fetch
21:36:23.409703 git.c:344               trace: built-in: git 'fetch'
21:36:23.486758 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:36:23.506772 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:36:23.506772 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://chromium.googlesource.com/chromium/src.git'
21:36:28.542596 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
21:36:28.578620 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
21:36:28.601635 git.c:344               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://chromium.googlesource.com/chromium/src.git/'
remote: Counting objects: 67598, done
remote: Finding sources: 100% (28/28)
21:39:29.787441 run-command.c:626       trace: run_command: 'unpack-objects' '--pack_header=2,28'
21:39:29.812462 git.c:344               trace: built-in: git 'unpack-objects' '--pack_header=2,28'
remote: Total 28 (delta 2), reused 20 (delta 2)
Unpacking objects: 100% (28/28), done.
21:39:30.083690 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
21:39:30.108704 git.c:344               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
From https://chromium.googlesource.com/chromium/src
   fd56d3ad0528..fbcc09282dc8  master     -> origin/master
21:39:30.788186 run-command.c:1452      run_processes_parallel: preparing to run up to 1 tasks
21:39:30.795191 run-command.c:1484      run_processes_parallel: done
21:39:30.813200 run-command.c:626       trace: run_command: 'gc' '--auto'
21:39:30.836212 git.c:344               trace: built-in: git 'gc' '--auto'


Bug:749709
Change-Id: I5b94549c42271885365c2354dc3f58399d6aebe0
Reviewed-on: https://chromium-review.googlesource.com/786691
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@google.com>
8 years ago
Andrii Shyshkalov ddfead3483 bot_update: mooore refactor around _git_checkout.
Bug:
Change-Id: I7141022348cdfd4dafbd2fa00b5981faa0ba79e0
Reviewed-on: https://chromium-review.googlesource.com/771930
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
8 years ago
Andrii Shyshkalov 75c0b4b715 bot_update: don't crash badly if there no json outpot.
To illustrate, with this CL it's clear what is broken:
  https://ci.chromium.org/swarming/task/39e5676c13f00710?server=chromium-swarm.appspot.com
while before you get a red herring:
  ncaught Exception: AttributeError("'NoneType' object has no attribute
  'get'",)...
  https://ci.chromium.org/swarming/task/39e55d46c0774610?server=chromium-swarm.appspot.com

R=iannucci@chromium.org

Bug: 786486
Change-Id: I9ed2cdb7261c2e22a5a68116b81f841ece06c5d2
Reviewed-on: https://chromium-review.googlesource.com/777556
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 8762632eb7 Reland "bot_update: make --output_json required in script invocation."
This is a reland of 331a276b54
Original change's description:
> bot_update: make --output_json required in script invocation.
> 
> It's already set by recipe, so this should have 0 difference in prod.
> 
> R=iannucci@chromium.org
> 
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

Change-Id: I165da9650df6b5e8d382aafc30914eff9a524c56
Reviewed-on: https://chromium-review.googlesource.com/777820
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Robert Flack fe36710a72 Revert "bot_update: make --output_json required in script invocation."
This reverts commit 331a276b54.

Reason for revert: Looks like this is causing bots to fail on the waterfall

Example:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F60913%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout

Original change's description:
> bot_update: make --output_json required in script invocation.
> 
> It's already set by recipe, so this should have 0 difference in prod.
> 
> R=​iannucci@chromium.org
> 
> Bug:
> Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
> Reviewed-on: https://chromium-review.googlesource.com/769164
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

TBR=iannucci@chromium.org,tandrii@chromium.org

Change-Id: I65d7665e45dde539f6436ffe8932319e03f96f25
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/777619
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov f151da9fac bot_update: refactor _git_checkout moving invariants out of the loop.
Bug:
Change-Id: Ie8781f73236e13c0afedf18dacf770d14563ab9f
Reviewed-on: https://chromium-review.googlesource.com/771857
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 331a276b54 bot_update: make --output_json required in script invocation.
It's already set by recipe, so this should have 0 difference in prod.

R=iannucci@chromium.org

Bug:
Change-Id: I1e1129b9748d6647e45eb26892c8e892019fe088
Reviewed-on: https://chromium-review.googlesource.com/769164
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov f78690b687 bot_update: refactor hiding ugly revision details into lower layers.
Bug:
Change-Id: Ie1caef2d58dce12bafbbec29fa4a5cc65076626c
Reviewed-on: https://chromium-review.googlesource.com/771856
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
8 years ago
Andrii Shyshkalov f062cf21e4 bot_update: refactoring improving variable names.
Change-Id: I36c64bb46b967af00adcf69f2e867fd03204d7c2
Reviewed-on: https://chromium-review.googlesource.com/771910
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
8 years ago
Andrii Shyshkalov 1f620f0550 bot_update: refactor giant git_checkout into two.
Change-Id: I2c023a538bfe62955238b39034ed7460286789e5
Reviewed-on: https://chromium-review.googlesource.com/771454
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov d7e241a22c bot_update: fix minor pylint issues.
Change-Id: I2f500bb8089039e689d5a989eb77f91374528164
Reviewed-on: https://chromium-review.googlesource.com/771381
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 26e338f40d Revert "Add luci_migration recipe module."
This reverts commit 2a1ca3f68b.

Reason for revert: to be replaced by recipe_engine runtime module
in https://chromium-review.googlesource.com/#/c/infra/luci/recipes-py/+/759677  

Original change's description:
> Add luci_migration recipe module.
> 
> R=​iannucci@chromium.org
> 
> Bug: 782460
> Change-Id: I022deb4d8e3cee8d48e1acccc78f8681490c72b9
> Reviewed-on: https://chromium-review.googlesource.com/757820
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

TBR=iannucci@chromium.org,tandrii@chromium.org

Change-Id: Ieb874ad42b939edb66e35db3cfed0b8b8746e70e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 782460
Reviewed-on: https://chromium-review.googlesource.com/759357
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Andrii Shyshkalov 2a1ca3f68b Add luci_migration recipe module.
R=iannucci@chromium.org

Bug: 782460
Change-Id: I022deb4d8e3cee8d48e1acccc78f8681490c72b9
Reviewed-on: https://chromium-review.googlesource.com/757820
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
8 years ago
William Hesse 3b74d3333e Dart: Change url of Dart repository in gclient recipe module Dart config.
Bug:
Change-Id: I031f5787a87a4bddbd3492912814476daf2c9b4f
Reviewed-on: https://chromium-review.googlesource.com/738118
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
8 years ago
Ryan Tseng bb85b3636b bot_update source manifest: fix test_api.py to match the actual output
The test output was incorrect, this matches what it actually should look like.

Bug: 772529
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: release_scripts
Change-Id: I1ca9ceeb6d6b6f55b2ac7c90667c6a9904d2a43b
Reviewed-on: https://chromium-review.googlesource.com/751694
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
8 years ago
Ryan Tseng 70dea4270e Reland 2 "bot_update recipe: Upload source manifest"
This relands commit 47b67c426b.
    
Also:
* Fix bot_update.py output of manifest
    
Bug: 772529,776299
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build

Change-Id: I31828ea5bfbc284623be8e553980f83d0889de92
Reviewed-on: https://chromium-review.googlesource.com/748308
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
8 years ago
Ryan Tseng 5d790d53ea Revert "Reland "bot_update recipe: Upload source manifest""
This reverts commit 47b67c426b.

Reason for revert: Bug in bot_update.py resource, breaks recipe roller.

Original change's description:
> Reland "bot_update recipe: Upload source manifest"
> 
> This reverts commit c3d1208d5c.
> 
> Also:
> * Instead of replacing "manifest", just add a new "source_manifest" 
> to the output JSON.  This allow transition without breakage.
> * Change the test api so test for recipe output changes.
> 
> The plan is to land this first, switch all downstream to "source_manifest",
> and then remove the original "manifest" key.
> 
> Bug: 772529,776299
> Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: release_scripts
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Reviewed-on: https://chromium-review.googlesource.com/731378
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org

Change-Id: I7a4ee904075e8b75b8a47f9ef0cd8a633af85a9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529, 776299
Reviewed-on: https://chromium-review.googlesource.com/748312
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
8 years ago
Ryan Tseng 47b67c426b Reland "bot_update recipe: Upload source manifest"
This reverts commit c3d1208d5c.

Also:
* Instead of replacing "manifest", just add a new "source_manifest" 
to the output JSON.  This allow transition without breakage.
* Change the test api so test for recipe output changes.

The plan is to land this first, switch all downstream to "source_manifest",
and then remove the original "manifest" key.

Bug: 772529,776299
Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/731378
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
8 years ago
Jao-ke Chin-Lee 3cf028ea0f [bot_update] Only kick in deadline after the first try.
If the revision has not yet propagated, then we never retry, because the amount of time to fetch HEAD already exceeds the deadline.

Bug: 771007
Change-Id: I91ac30b2b53b4f75729357b3298c52856a4d350e
Reviewed-on: https://chromium-review.googlesource.com/729328
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
8 years ago
Michael Achenbach c3d1208d5c Revert "bot_update recipe: Upload source manifest"
This reverts commit 037f9b80bb.

Reason for revert: The structural change to manifest breaks the recipe
tester:
https://cs.chromium.org/chromium/infra/recipes/recipes/recipe_roll_tryjob.py?q=recipe_roll_tryjob.py&sq=package:chromium&dr&l=149
https://crbug.com/776299

Original change's description:
> bot_update recipe: Upload source manifest
> 
> Bug: 772529
> Change-Id: I3f833740ddfbc71be42b140c99dcd4bbfeed54fe
> Reviewed-on: https://chromium-review.googlesource.com/706071
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,agable@chromium.org

Change-Id: Iba9653d73b7f117ca2f94619ae5da900cd0c8c98
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529,776299
Reviewed-on: https://chromium-review.googlesource.com/728059
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
8 years ago
Ryan Tseng 037f9b80bb bot_update recipe: Upload source manifest
Bug: 772529
Change-Id: I3f833740ddfbc71be42b140c99dcd4bbfeed54fe
Reviewed-on: https://chromium-review.googlesource.com/706071
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
8 years ago
Michael Moss 7de54ef0a2 Fix handling of potential empty gitiles result.
This fixes an issue introduced by
crrev.com/b40a45149a73a6b00e92fd1d2df4c0b1543d0005 where a 404 result
might return None contents.

R=agable@google.com, phajdan@google.com

Bug:774355
Change-Id: I993a029adb8d92cb27e043cd3d9b17cc8aee2432
Reviewed-on: https://chromium-review.googlesource.com/717937
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
8 years ago
Michael Moss b40a45149a Allow specifying alternative "success" codes in gitiles requests.
This is needed in the release recipes because there are times when we
want to check that a file is _not_ there yet (i.e. 404 response).

R=agable@chromium.org, dnj@chromium.org

Change-Id: Iedf646d6c73ebf898e90afd2009840b1e5c5b1ab
Reviewed-on: https://chromium-review.googlesource.com/709916
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
8 years ago
Andrii Shyshkalov 3f1ab05820 Fix gclient config of slave.DEPS and internal.DEPS.
R=smut@chromium.org

Recipe-Nontrivial-Roll: build
Change-Id: If90d6b172c3cd67ca57ad35345a9988e91da2345
Reviewed-on: https://chromium-review.googlesource.com/714776
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: smut <smut@google.com>
8 years ago
Vadim Shtayura 08049e2db2 Revert "git_cache: Remove locks"
This reverts commit c3eb3fa335.

Reason for revert: lots of "runhooks" failure everywhere

Example: https://build.chromium.org/p/chromium.linux/builders/Linux%20Builder/builds/92720

Original change's description:
> git_cache: Remove locks
> 
> These aren't in use, and the original problem they were
> meant to solve has been solved at the gclient.py layer 
> using resource locking:
>   https://codereview.chromium.org/2049583003
> 
> Bug: 773008
> Change-Id: I6609f39d7f15604e0bb3d742a41c4f9fec87a57a
> Reviewed-on: https://chromium-review.googlesource.com/707728
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,agable@chromium.org,phajdan.jr@chromium.org

Change-Id: I31d5fef94f39f3a9f97b9e59121073b1f433d11e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 773008
Reviewed-on: https://chromium-review.googlesource.com/711054
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
8 years ago
Ryan Tseng c3eb3fa335 git_cache: Remove locks
These aren't in use, and the original problem they were
meant to solve has been solved at the gclient.py layer 
using resource locking:
  https://codereview.chromium.org/2049583003

Bug: 773008
Change-Id: I6609f39d7f15604e0bb3d742a41c4f9fec87a57a
Reviewed-on: https://chromium-review.googlesource.com/707728
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
8 years ago
Michael Moss c634ba5705 Allow gclient solution URLs to be |None|.
This is supported by gclient and has the effect of treating the solution
as a purely local, standalone DEPS (i.e. not as a source repo to
checkout or process in any way). It also prevents the solution from
being added to the output of 'gclient flatten'.

R=agable@chromium.org, phajdan.jr@chromium.org

Change-Id: Ie11246ec7a74097b96ea6a26983a63b3fb3123a5
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/698271
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
8 years ago
Michael Moss 210ed0ebfe Expose new gclient.sync() arg in gclient.checkout().
R=phajdan.jr@chromium.org

Change-Id: I7313b8a26a5f2934e26491fc4b9403ba6cea0cee
Reviewed-on: https://chromium-review.googlesource.com/691017
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
8 years ago
Aaron Gable a16ccaf11a bot_update: use patch repo instead of project if it is mapped
This is a step away from patch_project, which was a holdover
from Rietveld. Instead of mapping "projects" (which are
poorly-defined and not guaranteed to be unique) to subpaths,
instead map repository urls (which are at least unique).

Eventually we may be able to compute this directly from DEPS
instead of hardcoding the mapping here, but at least this is
a step in the right direction.

Bug: 765633
Change-Id: Idd65984fc6edefcbedb0438d38c2338b10b7e8e5
Reviewed-on: https://chromium-review.googlesource.com/690776
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
8 years ago