Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Pylance does not like the ending bracket on its own line. As
`UploadAllSquashed` above uses inline type annotations, rewrite
`_UploadAllPrecheck`s type annotations inline as well.
Bug: None
Change-Id: I9b34cb4834530fffb9cb1788cd690f34d805f96f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333896
Commit-Queue: Michael Cui <mlcui@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This reverts commit 6b84fbfb20.
Reason for revert: Jumped the gun before final resolution for b/316921158
Original change's description:
> Remove ability to use_goma in autoninja (chromeos excepted)
>
> Won't run a build if use_goma=true and target_os!="chromeos".
>
> Bug: b/277197166
> Test: Manual tests and updates to unit tests.
> Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166
Change-Id: Iaf506a10dfdc1c1672b2927636adf5773edc67c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333176
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Won't run a build if use_goma=true and target_os!="chromeos".
Bug: b/277197166
Test: Manual tests and updates to unit tests.
Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Fix warnings with Python 3.8 like:
.../depot_tools/cpplint.py:3006: DeprecationWarning: invalid escape sequence \%
"""Logs an error if we see certain non-ANSI constructs ignored by gcc-2.
.../depot_tools/cpplint.py:4842: DeprecationWarning: invalid escape sequence \(
"""Retrieves all the text between matching open and close parentheses.
Change-Id: I99987e766cda98e2b8d8476f7900b9f99d58761e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5328187
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Providing --package flag allows recipes.py to skip git calls it makes
to find repo root.
Bug: b/323243527
Change-Id: If507fbf15459d7ea73670df202e16c344f9467a8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5263441
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
The `bootstrap` function within `populate` is shadowing the `bootstrap`
argument. This causes the `_ensure_bootsrapped` function to receive a
function pointer rather than passing along the boolean value.
Bug: 326116358
Change-Id: I6c03eb8bfbbc3385e1e9e653e43fdf9416cd244f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5310976
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Kyle Farnung <kfarnung@microsoft.com>
We aren't using --git-cache on bots anymore, so we shouldn't need to
point people to the CrOS fork. Everything else is merged upstream.
Bug: b/187784544
Change-Id: Ifd261ce509b9b501c49c2b567be17e4d9014913c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5311255
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: George Engelbrecht <engeg@google.com>
This shouldn't really happen, but who knows what users might
(accidentally) try to do.
Change-Id: Ic968f133693532a816b9bd3c8c5b97cba3261bc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5309411
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Sergey Frolov <sfrolov@google.com>
Commit-Queue: Sergey Frolov <sfrolov@google.com>
This will be used in cases where git workspace is unavailable.
Bug: b/323243527
Change-Id: I51056cfbac50f76a0de1b98cfec488d8f133e353
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5263021
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This reverts commit 1592a89c9f.
Reason for revert:
b/325856422
Original change's description:
> Remove ability to use_goma via autoninja.
>
> Doesn't run a build if use_goma is true, point to reclient docs. Turns
> off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
>
> Bug: b/277197166
> Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
> Reviewed-by: Ben Segall <bentekkie@google.com>
> Reviewed-by: Ramy Medhat <abdelaal@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166 b/325856422
Change-Id: Iaec83b254af002f420bfd2df1154ce94e18fac15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5308674
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: David Munro <davidmunro@google.com>
Reviewed-by: David Munro <davidmunro@google.com>
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory. Should allow concurrent builds against the same output
directory as much as ninja allows it.
Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
git cl upload prompts a user if they upload a particularly large change,
determined by the number of CCs added on the change. With the watchlists
analyzer and eventual removal of CCing watchers from git cl, this check
needs to be updated to work without knowing the number of CCs.
This change does two things:
1. Moves the check from git cl upload to a pan-project presubmit check.
2. Update the heuristic of a LSC from > 100 CCs to > 100 files modified.
Bug: 324562917
Change-Id: I6bd0db9fde942273acebc320c8b52e81b02bd42f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5296199
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This is a reland of commit 4d2864f3a1
The previous change initiated "self._diff_cache()" per file.
Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: b/323243527
Change-Id: I4733b6870935d4200cb32255214c57cdb55e84f0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286636
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Had some debugging logging in autoninjas test. Removed it. Lines were erroneously added in https://crrev.com/c/5272474 to address b/277197166
Lines initially added to debug a problem in during development. In the initial review I received a +1 CR with a comment requesting they be removed. I made the change, started the patchset upload, went to a meeting, forgot to enter the patchset comment (thinking I had done it before the meeting) and submitted the change without the patchset. Basically context switching got me 😅
Bug: b/277197166
Change-Id: I504569c713dbca7302988c39230caf9e9e6fa2c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5292094
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Doesn't run a build if use_goma is true, point to reclient docs. Turns
off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
Bug: b/277197166
Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
Reviewed-by: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Change is that racing directory is deleted atexit instead of being
cleaned up in a sweep at the start, whick wouldn't work anyway for the
actual use case the change was for (in-use racing directories would get
deleted, breaking concurrent builds).
Bug:b/324877315
Change-Id: I3baf25b07c13061e9b4c14a3a701ed9eceb9bbb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5288415
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
This reverts commit 3569608028.
Reason for revert: This includes a fix for crbug.com/324358728.
The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.
[1] https://codereview.chromium.org/228353003
Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 566c3772a0.
Reason for revert: Seems to be causing flaky failures in win-trace angle try builder, see b/323830218 for details.
Original change's description:
> Changing location of temp racing directory.
>
> Put temporary racing artifacts into
> .reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
> simultaneous builds against the same output directory.
>
> Bug: b/321554715
> Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: Ben Segall <bentekkie@google.com>
Bug: b/321554715
Change-Id: Ib6fa285bb435d7c0d82960633364669e6575d73b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286354
Commit-Queue: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
This reverts commit e98e458a9f.
Reason for revert: keep depot_tools CCing watchers while b/324466185 is open
Original change's description:
> Deprecate --bypass-watchlists flag
>
> CCing watchers is now handled by the Watchlists AyeAye analyzer on
> all Gerrit projects that use it, and not CCing watchers locally has
> no effect. Rely on that instead of logic in git cl.
>
> Bug: 41494851, 40242793
> Change-Id: I58fdf8c0a741a9189cccec3b3b56d0a8a1c44058
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5276660
> Reviewed-by: Scott Lee <ddoman@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: 41494851, 40242793
Change-Id: Ied75beb163b2628343d4dcac5819b5bd77854de8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5281700
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Gavin Mak <gavinmak@google.com>
This reverts commit 4d2864f3a1.
Reason for revert: This still doesn't scale well with large commit:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8756578662173359185/+/u/presubmit/stdout?format=raw
This is now a different instruction called repeatedly needlessly:
```
git -c core.quotePath=false diff -p --no-color --no-prefix --no-ext-diff f0b151b0e519ceb3efae2ad8951d8a9ce86e4fbd...HEAD --no-renames; cwd=D:\b\s\w\ir\cache\builder\win_presubmit\src
```
called for each affected files.
We probably want to cache the output for this too?
Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: b/323243527, b/324293047
Change-Id: Iffde46facefa9d2e3a0d3a9ab5f8e753c53ea6d4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272317
Auto-Submit: Arthur Sonzogni <arthursonzogni@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Put temporary racing artifacts into
.reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
simultaneous builds against the same output directory.
Bug: b/321554715
Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Aravind Vasudevan
Gavin Mak
Stephanie Kim
Josip Sokcevic
mlcui
Michael Savigny
Mike Frysinger
recipe-roller
Gaby Baghdadi
Junji Watanabe
Kyle Farnung
Chan Li
Takuto Ikuta
Patrick Meiring
Arthur Sonzogni