This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.
Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.
Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
A followup change will add support for change diff provided as user
input through stdin/file.
Bug: b/323243527
Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
issues.skia.org/issues/324072603: DEPS were renamed but the
corresponding gitlinks and gitmodules entry were not updated. Update
to check that
1. DEPS entries point to gitlinks (if SYNC)
2. .gitmodules entries match paths of gitlinks
Bug: 324118800
Change-Id: I4c97b54c1912712aef309584bebaa4ea3b24d5df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273762
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Other changes in this release:
7bdd89e5b2ba3e1a50dc63519934b391cd8d9061 Don't clear PYTHONPATH when executing python.
47bee0125c710a0765f995cb7830bb422a49a7d3 vpython: remove unused "virtualenv" from spec
8efd36d70716f5ae294c5ac8f554206a308b1615 vpython: increase MaxPrunesPerSweep to 10
5d5689a6f9decffb50a7a2a2b60128f951bfedc0 vpython: remove workaround when no spec is found
51f326e6c1b022a2cc2d2a1bcf11048a7012ea63 vpython: provide an empty spec if no default spec
181914db26ce3f0a549a7ee158ff70da7ef2ab15 vpython: move pkgs to luci-go
7c353081c7a6b2c77698962e57ba30c702027178 vpython: restructure vpython pkgs
e2768da6d2bb99660558f5b13daac496a9ed9061 cipkg: allow separating package generation with package build
Bug: 41493440
Change-Id: Ic3e8dfa706de918c3d5d896f44b911dd415b5f55
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5276597
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit db0dcbbe7c.
Reason for revert: `gclient sync` no longer works on my local checkout with this change due to the error `No such file or directory: 'vpython'.`.
Original change's description:
> Delete vpython, leaving only vpython3.
>
> Bug: 1376538
> Change-Id: Ia7795f75fd05db63f155c5feffcc43822fb45bb1
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5187892
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Commit-Queue: Brian Ryner <bryner@google.com>
> Reviewed-by: George Engelbrecht <engeg@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Chenlin Fan <fancl@chromium.org>
Bug: 1376538
Change-Id: I7c32eea1f8ae07644ab81a35213de5c68c64dfb6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5260042
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Andy Perelson <ajp@chromium.org>
Commit-Queue: Brian Ryner <bryner@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
No script can run on py2 (due to print syntax change), so those
checks are useless (will never be printed).
Change-Id: I9cc7bc8b2665dd09cc7f8939ba255956f96af88a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5262958
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is not enabled by default because it is new and should be
considered beta.
Bug: 780263
Change-Id: I7a4813c7ac4cef7441c5373dff4bb1289e7b99b6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5259981
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Robert Sesek <rsesek@chromium.org>
Turns on deps cache, and increases it's size, for mac developers. In
some cases, it shows good improvements on chrome builds (cutting 1-4
minutes off a full chrome build, once populated).
Bug: b/322801289
Change-Id: I9cd2b35b5429228c07b0703d6c3344963e4fb9e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5258563
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
This reverts commit 92b780b0bd.
Reason for revert: mac arm experiencing hang
Original change's description:
> [gclient] Enable parallel sync on ARM Mac by default
>
> I realized that gclient sync doesn't use all the cores on ARM Mac.
> This CL changes the default for ARM Mac to use all the available cores.
>
> Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
> Commit-Queue: Junji Watanabe <jwata@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Junji Watanabe <jwata@google.com>
Bug: 1427050
Change-Id: I512d1d1ef0034f12fc9054c5234ae688ec281857
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5259766
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
I realized that gclient sync doesn't use all the cores on ARM Mac.
This CL changes the default for ARM Mac to use all the available cores.
Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reduces resources used for local compilation in racing for windows.
Makes machine running the build more responsive during the build.
Testing seems to show it doesn't impact overall build performance, and might even improve it slightly.
Bug: b/309819402
Change-Id: I6f4f41887b165c753508a57fcaa1a1a6931feee3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254559
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
On my windows cloudtop, without deps cache a build of chrome took around 2130s, with it enabled (and populated) a build took around 1665s, which is about a 7-8 minute improvement.
Bug: b/320457907
Change-Id: I82232cad57152f1a10d1e262125290d54bf8ab33
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5249053
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
The current implementation of CheckInfraFreeze relies on timestamps
being preconverted to UNIX timestamps because zoneinfo doesn't work by
default on Windows. The necessary tzdata package has been added so that
zoneinfo will work for Windows, so switching to datetime and zoneinfo is
now possible and makes it easier to update the time range for freezes.
Bug: 1521396
Change-Id: I8db111f05db2e79f4fffd9da3829b9071d12163f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5246425
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Importing pipes gets a DeprecationWarning. shlex.quote is equivalent to
pipes.quote. Use that instead.
Bug: 1522874
Change-Id: I18b8598f2e961d58ba71d886921bda9d9ad956ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5243707
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
This CL updates the warning printed when more than 10 CLs will be
created to be shown whether or not the CLs are being sent to the CQ
and provides guidance on adjusting the number of CLs using the max-depth
flag. This makes the tool more user friendly by preventing a new user
from unknowingly creating many CLs without realizing there is a
supported option to customize the depth.
Additionally, it sets the default for the cq-dry-run option to False
instead of None and logs the value of cq_dry_run when running a git cl
split dry run.
Change-Id: I6feba24d06dd5e88d9005345d5b0a1e15c53e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5234543
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Mike Dougherty <michaeldo@chromium.org>
The node config is an alias of node-ci, which builds upstream Node.js
with bundled V8 instead.
This config will be used by maintainers of Node.js to test the GN build
of Node.js.
Bug: v8:13803
Change-Id: Ib67c54a24c5f8aee79d8f6eab50a9b697f28daeb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5232378
Reviewed-by: Victor Gomes <victorgomes@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This will provide a single site that needs to be updated when setting
new dates for infra freezes rather than requiring copy-pasting timestamp
changes between multiple directories and repos.
Bug: 1515149
Change-Id: I62f7de45c3a7e993f558738a0ff1f2dae6cac471
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5160597
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
All git path parameters support `~` expansion, there's no reason we
shouldn't do this for cache.cachepath parameter used by gclient.
Change-Id: I92485d5f17d8ab8125074253f75e0ef2e2ff41e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5227662
Commit-Queue: Aleksey Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
It's possible that HEAD points to a different branch, and old one was
deleted. This ensures that cache still has the correct HEAD information.
Change-Id: Iab0b6badfd1f3e2c0e09fe542ee603b78d4a0041
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5218257
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Update the warning message for linux and mac goma builds, and add a 5
second delay.
Do not submit until Jan 22, 2024
Bug: b/277197166
Change-Id: Id93822b1aa32b05746b8fad4b9dc332cf61d291c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5215930
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
This is for some internal infrastructures which want to use service
accounts from corp machines.
Bug: b/320639792
Change-Id: Ie90241fe89e0bb0fb5526844e2e3651cc316d945
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5226384
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Richard Wang <richardwa@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
git_cache populate can get interrupted midway (e.g. by LUCI CV sending a
signal that build is no longer needed). When that happens, a git mirror
may be in a state where some commits are available, but cloning such
repositry results in an empty repsitory.
This change ensures that the initial fetch operation finished
successfully.
R=ddoman@google.com
Bug: 1517944
Change-Id: I1ee860860877dbfff7a444b45fe4515fe26b248c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5219032
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Other places in gclient_scm (GitWrapper._Clone, GitWrapper.update) do
the same.
Note that we call _AutoFetchRef even after running _Fetch, because
_Fetch in not guaranteed to fetch a specific revision. For example, a
checkout might be configured without branch heads and without tags, yet
a revision might point to a tag.
Change-Id: I5cfd8ebafa4b45afd72b0251f04e5d713199f2fd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4715239
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
+ Change scm GetAllFiles back to returning all files
+ should be safe to do since it's only used by presubmit code
+ add ignore_submodules option to CaptureStatus
+ It's used by other parts of depot_tools code
+ Add AffectedSubmodules to return submodules only
+ AffectedFiles still only returns non-submodule files.
Bug: 1475770
Change-Id: I457ad96099fb20eff4cc1a4fc84f59e7ae707abe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5216005
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Change-Id: Ia37291aca59b429b59bdc6424a2f36394d558e3f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5214265
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Update use_goma warning for linux and mac, to add date of removal of
use_goma (Feb 7th) and a warning that delays will be added (Jan 22).
Bug: b/277197166
Change-Id: Id567e5bf704c5d80e5122663c6d8194ac510d992
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5200850
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
This uses `re` instead, which should have similar performance with python3.
Bug: 1517826
Change-Id: If03f386d26590dcae7da6b4f27ec1ea4926065fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5197707
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>
A recent change to autoninja breaks 'compile_single_file' because
autoninja expects to be able to find the 'google.auth' package, which is
available via vpython3.
Change-Id: Ie0c9bc229108daadae27bd85ffc0471d9e7e7fca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5199126
Commit-Queue: Sébastien Marchand <sebmarchand@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
The version we are using now does not run under python 2.7.
Add an explicit python_version 3.8 to the inline spec so that we don't
rely on vpython's default version. This is a no-op change when running with vpython3. We should run yapf under 3.11, but this needs additional testing and likely an update of zipp, so we'll do it separately.
Change-Id: If1fccbe6b13c77a770bd1a8c9de0789d543ba890
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5198109
Auto-Submit: Brian Ryner <bryner@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
siso binary will print json etc (e.g. `siso fs export`)
and usually used to pipe to jq or so.
but siso.py print message when $SISO_PATH is set
and corrupt json message.
use stderr for such message.
Change-Id: Ic35ecabc159fb2053a8a29816c276d914f8d2468
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5197566
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Fixed https://crrev.com/c/5091007
This change includes commits:
0c1a41fa62950eedcaaee58ba7268d3813871e3f vpython: preserve NoDefaultCurrentDirectoryInExePath state
a7a2cf379c762dc22a35fc4e1af9e4dfd8df2785 vpython: remove unused code
65fb8069dc290e3b3cc4765f57a78fbbe0ed2093 vpython: update script used for gathering changes
dca9f0e513fae6d176bd264766dec6c464c38e8e Run compileall in parallel.
a76e08f54c62e239b37f020040382f4918d9e056 cipkg: set NoDefaultCurrentDirectoryInExePath on windows
a4b9b9d8f8ba904212a9ff3a33be76060d380f3c cipkg: append $out to derivation envs
445bf8af01dafa28d86e8aedac9ff091a4c64662 cipkg: add peep-software-deployment as owner
50b7bd75850311895fac95eb602df392d982345f Split the global linter config into per-directory configs.
00e23cbde55df05e0e42f7a574c79ff8bcf589c4 vpython: match tag before verify
ef99f244101ffd1b549c740ef6059b255bdbd49c fix gci-ed warning for vpython and pkgbuild
ba2a688d3175807a5f66ab64afd649c867cf563a vpython: rename vpython_ng to vpython
Change-Id: Iafbbbbd4408f62436cf6af17fe9b86cf5d66d06e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5191526
Commit-Queue: Chenlin Fan <fancl@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>
The previous binary was so old that it does not work on recent
versions of macOS.
Note that `cipd selfupdate-roll` requires the cipd package to exist
at the given tag for all platforms that the package has ever been
built for. Therefore, I'm using a revision after the latest platform
was added to the builders.
Change-Id: I865c31a5e3823ffc344bcb6585a29935ef6214a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5191588
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Brian Ryner <bryner@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
With this change, python2[.bat] will no longer be available in
depot_tools root. vpython continues to work.
Bug: 1376538
Change-Id: Iabe22e4e33d0d79d258068c90128da8de4d3ece1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5170265
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Aravind Vasudevan
Gavin Mak
Josip Sokcevic
Brian Ryner
recipe-roller
Rupert Wiser
Robert Sesek
Michael Savigny
Scott Lee
Junji Watanabe
Garrett Beaty
Mike Dougherty
Joanna Wang
Cheng Zhao
Aleksey Khoroshilov
Alexei Svitkine
Takuto Ikuta
Nikita Leshenko
Joanna Wang
Michael Achenbach
Sebastien Marchand
Fumitoshi Ukai
Chenlin Fan