R=sokcevic
Rationale: both of us are member of source team now and we will
need to review CLs from contributors during our oncall rotation
Change-Id: If7a5f3ece668df0ed799653707f7e8378e753f56
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4766323
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This ensures that for ninja+reclient builds we still start reproxy to allow `rewrapper clang ...` to run locally.
Also this fixes a preexisting bug where RBE_remote_disabled=1 did not apply to the ninja call when running on mac
Bug: b/295192141
Change-Id: I37c4ffdd5f30d423b84f6c469f5edfa88efe1344
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4659225
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
The `--format` option for `git ls-tree` was introduced in git v2.36 and
hence gclient breaks for lower git versions.
Change-Id: I02750904aa78bb5fd5f3f57dc2ad42e355fa4a42
Bug: 1471241
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4765682
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Run it in a subshell to avoid forking+execing another shell.
Bug: 1469696
Change-Id: I173f18aa3bf2f181db261750f4e51422717886ce
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4753234
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
The ParseGitModules() uses `git submodule status` result to get the
commit hashes for the submodules. It assumes the command's result to not
have a `git describe` at the end of each line, and hence only works in
certain scenarios. This change fixes this by updating the parser.
Change-Id: I8f4e3fae6562dafd0e127255a8f26e7336a75683
Bug: 1471241
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4763348
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit e121d14b12.
Reason for revert: git for mac arm is widely available now so
this hack is no longer needed. See: b/228226478
Original change's description:
> [cipd] Detect Mac ARM running under Rosetta
>
> If a user has git for x64, `git cl` results in downloading x64 cipd
> packages. Other operations, such as gclient or git-cl will result in
> downloading arm64. This flip-flopping between architectures causes
> unexpected behavior:
> (1) slows down commands since packages are wiped and new ones are
> downloaded.
> (2) long running processes, such as goma may crash.
>
> This change adds additional check for all macs running on x86 to check
> if is running under Rosetta and sets ARCH=arm64 if true.
>
> R=bryner@google.com, masonf@chromium.org
>
> Bug: 1311733
> Change-Id: I65b127467d5c47d5bf07952d2ecdcb93630c7c87
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3573068
> Reviewed-by: Brian Ryner <bryner@google.com>
> Reviewed-by: Mason Freed <masonf@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1311733
Bug: b/228226478
Change-Id: I3888769944d7fe200a0bff8cf0a1c04407d50bdf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4763444
Reviewed-by: Jonathan Nieder <jrn@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
autoninja is intended to mean "build this directory, don't bother me
with the details". As we transition to siso it is therefore appropriate
that it should take on the duty of deciding whether to invoke ninja or
siso.
By looking for a use_siso gn arg autoninja can easily do this.
This change relies on crrev.com/c/4753433 to add the use_siso gn arg.
This change also teaches autoninja to detect if a user switches
between siso and ninja without doing a gn clean inbetween.
Note that this change also teaches autoninja to invoke autosiso or
siso ninja based on whether use_remoteexec is true.
Bug: b/293657720
Change-Id: I3ad36a81857e75ffe6babc4f107b777e733a285b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4749722
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
`Developer builds with reclient are not yet supported` is misleading.
This CL updates the message to help users to resolve the issues.
Bug: b/294933473
Change-Id: Ibce2854e25b8994af58011f4e7661f467fe910e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4755000
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Detect failure of scm.GIT.RefToRemoteRef and raise an error before
attempting to join its results as a string. This makes certain
repository errors diagnosable.
Fixed: 1470430
Change-Id: I19cd86d9ed088f5cd511e4c9d7dbe5844165c7cf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4754237
Auto-Submit: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
On NOP or incremental builds the reproxy startup/shutdown times can be a
significant portion of the total build time and it is important to
acknowledge these times.
Bug: b/294443556
Change-Id: Ic63331683632c8f447858b498f53d254b56ffe89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4753472
Reviewed-by: Philipp Wollermann <philwo@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This info is only used when bootstrapping for the first time,
so move these vars to that function to avoid wasted effort.
Bug: 1469696
Change-Id: I21b97b2429cf2dac6242e2e86aa7668b09ed47c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4752396
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Avoid recomputing this value multiple times between vpython & cipd.
Bug: 1469696
Change-Id: If82d58180f94cb2833a5a177f7b1e431e08f2892
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4750040
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
This reverts commit 43083529de.
Reason for revert: crbug.com/1470122
Original change's description:
> New vpython to depot_tools
>
> This is a reland of https://crrev.com/c/4653897
> Fixed the issue for cipd wrapper. Now all environment variables should
> be perserved when invoking cipd: https://crrev.com/c/4669637
>
> 1. virtualenv field in the spec is ignored.
> 2. --vpython-tool removed support for delete and help subcommands.
> 3. --vpython-tool installed removed support for naming venv.
> 4. removed support for -vpython-interpreter.
> 5. removed support for searching interpreters in host PATH.
> 6. python 2.7 is available only if the binary is invoked as `vpython`.
> 7. fixed a bug that passes invalid vpython arguments to the script,
> which may be silently ignored.
> 8. python_version in the vpython_spec must specify a minor version
> (python_version: "3" is not valid anymore).
> 9. vpython now requires the cipd binary to be present in PATH, which is
> true already when using Swarming or depot_tools.
>
> Also updates the LUCI_OWNERS to add peep-software-deploy team.
>
> Bug:1415212
> Change-Id: Ie541a2a60bef829a976a13db9a6732b406c4d878
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4719827
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> Auto-Submit: Chenlin Fan <fancl@chromium.org>
> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Bug: 1415212
Change-Id: I6ca32066acd977a293f8b8f42697c383cc2a93fc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4751179
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Chenlin Fan <fancl@chromium.org>
A lot of scripts like to compute the dirname of $0 to find itself to
load more files. Have some scripts export & resuse this value so we
avoid recalculating when not needed.
Bug: 1469696
Change-Id: I3aac2e6119dbfa3c68715208df2786f67cdce92e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4750430
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Other depot_tools scripts use dirname regardless of mingw,
so do that here too to simplify.
Bug: 1469696
Change-Id: Ia3c22a4eb72e62564eee1e8b18a88671c8b8b9a5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4748852
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
The chromium-style plugin was replaced last year with cl/495068108.
Change-Id: Ifce7cf5e22f10681ffe743572182d759ae51fed2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4743665
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Other depot_tools scripts use dirname regardless of mingw,
so do that here too to simplify.
Bug: 1469696
Change-Id: I9a70975ef5614b1107ac87a5dc29b5ee1a06e298
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4750039
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
This was added long ago to workaround svn behavior with symlinked dirs.
We haven't used svn in years, so drop this logic to simplify.
Bug: 1469696
Change-Id: Ic881ec74b1a3caa002a131be48259a56be30d4f4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4745419
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
This CL stops "git cl split" from printing which directories could
not be uploaded if upload fails. This information is redundant with
previous lines printed by "git cl split"
Bug:None
Change-Id: I39bc47e3b29d4ce777a65b6bdac28b19b672339f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4743774
Commit-Queue: Peter Kotwicz <pkotwicz@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
When running on macOS under python3 as part of PRESUBMIT_test.py,
a ResourceWarning is printed. This is due to the `open(..).read()`
idiom.
Use a `with` statement to ensure the file is properly closed.
Bug: none
Change-Id: I5643db0b77e6896b4b86177d2c1fe8dea04b58ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4742422
Reviewed-by: Gavin Mak <gavinmak@google.com>
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This is a reland of https://crrev.com/c/4653897
Fixed the issue for cipd wrapper. Now all environment variables should
be perserved when invoking cipd: https://crrev.com/c/4669637
1. virtualenv field in the spec is ignored.
2. --vpython-tool removed support for delete and help subcommands.
3. --vpython-tool installed removed support for naming venv.
4. removed support for -vpython-interpreter.
5. removed support for searching interpreters in host PATH.
6. python 2.7 is available only if the binary is invoked as `vpython`.
7. fixed a bug that passes invalid vpython arguments to the script,
which may be silently ignored.
8. python_version in the vpython_spec must specify a minor version
(python_version: "3" is not valid anymore).
9. vpython now requires the cipd binary to be present in PATH, which is
true already when using Swarming or depot_tools.
Also updates the LUCI_OWNERS to add peep-software-deploy team.
Bug:1415212
Change-Id: Ie541a2a60bef829a976a13db9a6732b406c4d878
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4719827
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Chenlin Fan <fancl@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
This CL prevents split_cl.py from uploading multiple CLs for the same
reviewer set.
BUG=1468350
Change-Id: I9c328589f7facfe10ee5066cc3d1cda007dd1d2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4726781
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Peter Kotwicz <pkotwicz@chromium.org>
CrOS intends to provide a Bazel executable for our users in
chromite/bin/bazel in our tree. We'd like the "bazel" command in
depot_tools to call this executable.
This adds a new launcher to depot_tools which searches for that bazel
executable when located inside of a ChromiumOS checkout, and executes
it. When located outside of a ChromiumOS checkout, this launcher
"disappears", searching elsewhere in the PATH for another Bazel
executable.
Since other teams using depot_tools may want to start using Bazel in
the future, this launcher is intended to have shared ownership: other
teams are welcome to come add their search functions to the launcher
if they require the same functionality as us.
Bug: b:253268519
Change-Id: I61f6383d8b69b9eea622f37277678f898cc7fd6b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4718785
Reviewed-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aaron Massey <aaronmassey@google.com>
Commit-Queue: Jack Rosenthal <jrosenth@chromium.org>
Auto-Submit: Jack Rosenthal <jrosenth@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This makes hooks path behavior match deps path behavior.
Bug: 1468658
Change-Id: Ib6685cb52679249b389599ada549e2c4410e812c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4729084
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
On macOS the default limit on open file descriptors is really low
and causes ninja to fail with 'Too many open files' if run with a
value of -j greater than 200.
Running only 200 jobs however slow down the build considerably so
check whether the limit has been raised by the user or if it can
be raised programmatically. In the positive case, use a limit of
800 jobs, otherwise settle down on 200 jobs.
This should allow running ninja with a large number of jobs even
on macOS Ventura 13.5 which now requires to set the limit both in
/Library/LaunchDaemons/limit.maxfiles.plist and via ulimit -n.
Bug: 1467777
Change-Id: Ib8b7d0d1ee47d243c1872229c5340e7795c1b42e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4725183
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
The limit set in /Library/LaunchDaemons/limit.maxfiles.plist appears
to no longer be respected in macOS Ventura 13.5 and the OS forces a
limit of 256 file descriptors, causing build with autoninja to fail
on macOS running this most recent version of the OS.
Force the max number of process to 200 until a new working way to
increase the file descriptor limit is found. This will allow devs
working on macOS to be able to build (albeit slower than before).
Fixes the following build failure when goma is enabled and running
on macOS Ventura 13.5:
$ autoninja -C out/Debug-iphonesimulator chrome
ninja: Entering directory `out/Debug-iphonesimulator'
[0/3574] CXX ....oninja: fatal: pipe: Too many open files
Bug: 1467777
Change-Id: Ia7eaab552f7e6d26a2f48d72bb8235a70d6d442f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4720227
Reviewed-by: Dirk Pranke <dpranke@google.com>
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
local_resource_fraction is set to 0.2 as goma limited the max local
racing subprocesses to num_cpu/5
racing_bias is set to 0.95 as experimentally that ensures reasonable
behaviour for cache hits.
This cl does not enable racing by default. These values are only used if
RBE_exec_strategy=racing is set
Test: Tested builds on my mac and linux cloudtop and got mostly remote exec
and cache hits with some local executions
Bug: b/288564971
Change-Id: Idb71c03f4addd23d0253dc5bf3564058cd212284
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4713186
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Joanna Wang
recipe-roller
Yiwei Zhang
Ben Segall
Aravind Vasudevan
Mike Frysinger
Josip Sokcevic
Chan Li
Bruce Dawson
Junji Watanabe
Kenneth Russell
Arthur Milchior
Chenlin Fan
Gavin Mak
Peter Kotwicz
Vadim Shtayura
Sylvain Defresne
Jack Rosenthal
Struan Shrimpton
Anne Redulla