Commit Graph

86 Commits (79cfa048c0e66e7a9478f4e993aadd3873cc1fd9)

Author SHA1 Message Date
L. David Baron 79e1a7174c autoninja: Show additional help information for --help, like for -h.
This doesn't catch all the variants of --help that getopt allows but it
does catch --help, which is probably the most common.

Bug: 329317224
Change-Id: I2dde63df84bd2c2ade5f9d2c1ab10c3dc13fba70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370829
Auto-Submit: David Baron <dbaron@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
12 months ago
Michael Savigny beb48f193f Minor change to -h output for autoninja.
Notes that --offline is for disabling remote execution in general not just for goma.

Bug: b/329317224
Change-Id: I5538c9fac6d5161e98e1d156e61c512c6a1e8496
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370094
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
12 months ago
Michael Savigny 50de666ba4 Revert "Remove ability to use_goma in autoninja (chromeos excepted)"
This reverts commit 6b84fbfb20.

Reason for revert: Jumped the gun before final resolution for b/316921158

Original change's description:
> Remove ability to use_goma in autoninja (chromeos excepted)
>
> Won't run a build if use_goma=true and target_os!="chromeos".
>
> Bug: b/277197166
> Test: Manual tests and updates to unit tests.
> Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>

Bug: b/277197166
Change-Id: Iaf506a10dfdc1c1672b2927636adf5773edc67c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333176
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
1 year ago
Michael Savigny 6b84fbfb20 Remove ability to use_goma in autoninja (chromeos excepted)
Won't run a build if use_goma=true and target_os!="chromeos".

Bug: b/277197166
Test: Manual tests and updates to unit tests.
Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta 280bb93210 Revert "Remove ability to use_goma via autoninja."
This reverts commit 1592a89c9f.

Reason for revert:
b/325856422

Original change's description:
> Remove ability to use_goma via autoninja.
>
> Doesn't run a build if use_goma is true, point to reclient docs.  Turns
> off use_goma for linux and mac builds.  Windows will follow in 2-3 weeks.
>
> Bug: b/277197166
> Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
> Reviewed-by: Ben Segall <bentekkie@google.com>
> Reviewed-by: Ramy Medhat <abdelaal@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>

Bug: b/277197166 b/325856422
Change-Id: Iaec83b254af002f420bfd2df1154ce94e18fac15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5308674
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: David Munro <davidmunro@google.com>
Reviewed-by: David Munro <davidmunro@google.com>
1 year ago
Michael Savigny 1592a89c9f Remove ability to use_goma via autoninja.
Doesn't run a build if use_goma is true, point to reclient docs.  Turns
off use_goma for linux and mac builds.  Windows will follow in 2-3 weeks.

Bug: b/277197166
Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
Reviewed-by: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 26b100def9 Update goma deprecation message and add delay to build start for goma.
Update the warning message for linux and mac goma builds, and add a 5
second delay.

Do not submit until Jan 22, 2024

Bug: b/277197166
Change-Id: Id93822b1aa32b05746b8fad4b9dc332cf61d291c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5215930
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta 3a7f62489a autoninja: make it possible to skip external account check
This is for some internal infrastructures which want to use service
accounts from corp machines.

Bug: b/320639792
Change-Id: Ie90241fe89e0bb0fb5526844e2e3651cc316d945
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5226384
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Richard Wang <richardwa@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Michael Savigny 83458774b1 Update autoninja use_goma warning message.
Update use_goma warning for linux and mac, to add date of removal of
use_goma (Feb 7th) and a warning that delays will be added (Jan 22).

Bug: b/277197166
Change-Id: Id567e5bf704c5d80e5122663c6d8194ac510d992
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5200850
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Michael Savigny f26dddb204 Add use_goma deprecation warning to autoninja.
Adds the same warning to every use_goma build that is currently printed
on a gn gen with use_goma=true.

Bug:b/277197166
Change-Id: Icf9781f814416dfbeb2826fc88372465155fcd2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5123231
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta aa65be9856 autoninja: cache negative result of gcloud account check
This is to remove 1~3 seconds from no-op build using autoninja.

Bug: b/309720176
Change-Id: I2c15a43517d5a99eae794015c18c884a3f3fdef4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5092847
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
1 year ago
Takuto Ikuta cccca95d8f Revert^2 "autoninja.py: disallow external account from corp machine"
This reverts commit ed596dadf9.

Reason for revert:
Ignore exception when getting account for application default
credentials.

Original change's description:
> Revert "autoninja.py: disallow external account from corp machine"
>
> This reverts commit 0d0f28a4fc.
>
> Reason for revert:
> http://b/309720176#comment7
>
> Original change's description:
> > autoninja.py: disallow external account from corp machine
> >
> > This disallows Googler to use non-google account with reclient and
> > siso from corp machine.
> >
> > Bug: b/309720176
> > Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
> > Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> > Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
>
> Bug: b/309720176
> Change-Id: I62b4891b1e2e2554d088d162057982b95c8adcd7
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065849
> Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> Owners-Override: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Bug: b/309720176
Change-Id: Ib9aef1b7fa8e3c9bb20ac8d6b461c6bcff72dc83
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065052
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Takuto Ikuta ed596dadf9 Revert "autoninja.py: disallow external account from corp machine"
This reverts commit 0d0f28a4fc.

Reason for revert:
http://b/309720176#comment7

Original change's description:
> autoninja.py: disallow external account from corp machine
>
> This disallows Googler to use non-google account with reclient and
> siso from corp machine.
>
> Bug: b/309720176
> Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
> Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>

Bug: b/309720176
Change-Id: I62b4891b1e2e2554d088d162057982b95c8adcd7
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065849
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Owners-Override: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
1 year ago
Takuto Ikuta 0d0f28a4fc autoninja.py: disallow external account from corp machine
This disallows Googler to use non-google account with reclient and
siso from corp machine.

Bug: b/309720176
Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Takuto Ikuta df3e577855 apply black and `git cl format` for ninja related python files
This is made by
$ black --line-length 79 *ninja*.py
$ git cl format

Change-Id: Ic446898a5461ae536542f6312cae2ce126dfe82a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035265
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
1 year ago
Takuto Ikuta 5cbc521beb autoninja.py: remove use_rbe support
This arg was not used anymore.

Change-Id: If578845fc3e200f4f23c8ca050542b92fcf13ef2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035261
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
1 year ago
Philipp Wollermann 17debef507 autoninja.py: Enable offline mode for Siso when use_remoteexec not set
Tested with:

$ autoninja -C out/fastbuild-siso-local base
offline mode
 2.48s loading build.ninja
pre:0 local:176 remote:0 5.7/s fallback:0
[...]
5m40.86s Build Succeeded: 1641 steps - 4.81/s

$ autoninja -C out/fastbuild-siso-reclient base
Running gcloud auth print-access-token --trace-token=interactiveauth
Proxy started successfully.
124.260 s to start reproxy
 5.32s loading build.ninja
15.93s Regenerating ninja files
 2.75s update filegroups
 4.84s loading build.ninja
pre:0 local:1 remote:0 30.2/s cache:100.00% fallback:0
[...]
2m05.55s Build Succeeded: 2560 steps - 20.39/s

R=jwata, ukai

Change-Id: Ib976b874796a20dca55005986ee6f7ee3c942245
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4997466
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
1 year ago
Fumitoshi Ukai 406be8281e autoninja: don't allow use_siso=true and use_goma=true
Bug: b/306048338
Change-Id: Ib34eb5abe3c419d4afefb7a9cd083441ef755110
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4951553
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
1 year ago
Junji Watanabe c51829968b [autoninja] Update Siso .ninja_log comments
Siso is going to generate non-empty .ninja_log for the analysis tools.
https://crrev.com/c/4907677

This CL fixes the comment about empty ninja_log.

It's not compatible with Ninja's .ninja_log. So `gn clean` will still
be required.

Bug: b/298594790
Change-Id: Ib4c60f3ed22f516d6f7e2847aaf57e228121eccf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4937692
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Philipp Wollermann 0b943400a4 Execute Ninja / Siso directly from autoninja.py.
Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells.

Part of my bigger auto{ninja,siso} refactoring.

Tested:
- Handling of the ^^ suffix on Windows still works correctly.
- Handling of error codes - i.e.; making sure
  "autoninja base_unittests && base_unittests.exe" behaves properly
  in the success/failure case.
- Make sure the command prompt title is reliably reset on exit.

I tested autoninja with all combinations of these:
- Host platform: Linux, macOS, Windows
- Remote GN args: <none>, use_goma=true, use_remoteexec=true
- Siso GN args: <none>, use_siso=true
- Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows)

R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org

Bug: b/293657720
Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
1 year ago
Henrique Ferreiro 7eb4e4841f Don't limit ninja -j on Linux beyond ulimit -n
macOS limits the ninja j value to 1000, because ninja has a limit to the
number of open file descriptors of FD_SETSIZE, which is 1024 on Darwin.
On Linux, the ninja binary distributed on Chromium seems to be compiled
with poll.h support, so that this limitation doesn't exist:
22b778ca19/src/subprocess-posix.cc (L59)

Change-Id: I97848bb99c08fe118dbdaea525da713382373c9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4866223
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
1 year ago
Samuel Attard 8a25998dd9 handle absolute and relative gn imports in autoninja
Follow up to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864898. Now
that autoninja attempts to parse and follow import directives we
also need to support relative and absolute paths not just project
(//*) references.  This adds supports for those imports in
the added autoninja logic.

Change-Id: I93209b3db76e4e3d296616a3aace32e4d5fc5109
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4869108
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Samuel Attard <samuel.r.attard@gmail.com>
1 year ago
Bruce Dawson 30c1cba02f Handle import statements in args.gn files
Did you know that args.gn files can have import statements and
conditionals? I did not, but apparently some developers make use of both
of these.

Supporting import statements is not too hard, so this change adds this
support. Supporting conditionals is possible, but risks turning
autoninja into a turing complete language which is more than I think we
want to do.

This doesn't use the similar code in tools/mb/mb.py because that
code is complex, and relies on the script location to find the src
directory.

This change also updates two of the existing test conditionals that
were not quite sufficient - ninja/autoninja default to num-cores
plus 2 so > cpu_count() is actually not sufficient to prove
anything.

Bug: 1482404
Change-Id: I0539d8068af59d11927cbdad260278a24ab912e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864898
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
1 year ago
Henrique Ferreiro 8bde16478f Improve autoninja resource limit handling
This CL makes autoninja raise the number of allowed open file
descriptors to its hard limit, instead of a random value.

Also, independently of whether this operation was successful or not, the
ninja j value is capped at 80% of the current value of allowed open file
descriptors. This approximately matches the previous value of 200 for
the default value of 256 in macOS, but it's also more future-proof.

Lastly, this changes are also applied on Linux.

Change-Id: Idf2cd08384fe9f2bc699293d7062122590284dba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4852717
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
1 year ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Bruce Dawson 9c4fbc5a2a Improve ninja-to-siso switch detection
autoninja tries to detect if a user switches an output directory from
ninja to siso without doing a "gn clean" between. Initiallly this
detection looked for .ninja_deps, but this file doesn't always exist
after a ninja build. This change switches to looking for .ninja_log.

However autosiso creates a .ninja_log file (currently zero length) so
the new check is:

a) If .siso_deps exists then a siso build has been done and only siso
builds are allowed.
b) If .siso_deps doesn't exist but .ninja_log does then a ninja build
has been done and only ninja builds are allowed.
c) If neither file exists then the directory was cleaned and any type of
build is allowed.

Bug: b/293657720
Change-Id: I2cdba74f0894e62aa0e68f91e225f497b2425f45
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4784103
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
2 years ago
Ben Segall 467991e727 reclient: support -o and --offline flags in autoninja
This ensures that for ninja+reclient builds we still start reproxy to allow `rewrapper clang ...` to run locally.

Also this fixes a preexisting bug where RBE_remote_disabled=1 did not apply to the ninja call when running on mac

Bug: b/295192141
Change-Id: I37c4ffdd5f30d423b84f6c469f5edfa88efe1344
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4659225
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Bruce Dawson 5a4c350ee9 Teach autoninja to handle siso/autosiso
autoninja is intended to mean "build this directory, don't bother me
with the details". As we transition to siso it is therefore appropriate
that it should take on the duty of deciding whether to invoke ninja or
siso.

By looking for a use_siso gn arg autoninja can easily do this.

This change relies on crrev.com/c/4753433 to add the use_siso gn arg.

This change also teaches autoninja to detect if a user switches
between siso and ninja without doing a gn clean inbetween.

Note that this change also teaches autoninja to invoke autosiso or
siso ninja based on whether use_remoteexec is true.

Bug: b/293657720
Change-Id: I3ad36a81857e75ffe6babc4f107b777e733a285b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4749722
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Sylvain Defresne 7b4ecc7c35 [apple] Raise max number of open file descriptors if possible
On macOS the default limit on open file descriptors is really low
and causes ninja to fail with 'Too many open files' if run with a
value of -j greater than 200.

Running only 200 jobs however slow down the build considerably so
check whether the limit has been raised by the user or if it can
be raised programmatically. In the positive case, use a limit of
800 jobs, otherwise settle down on 200 jobs.

This should allow running ninja with a large number of jobs even
on macOS Ventura 13.5 which now requires to set the limit both in
/Library/LaunchDaemons/limit.maxfiles.plist and via ulimit -n.

Bug: 1467777
Change-Id: Ib8b7d0d1ee47d243c1872229c5340e7795c1b42e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4725183
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
2 years ago
Sylvain Defresne 4d992437eb [apple] Set limit of 200 process when running on macOS Ventura 13.5
The limit set in /Library/LaunchDaemons/limit.maxfiles.plist appears
to no longer be respected in macOS Ventura 13.5 and the OS forces a
limit of 256 file descriptors, causing build with autoninja to fail
on macOS running this most recent version of the OS.

Force the max number of process to 200 until a new working way to
increase the file descriptor limit is found. This will allow devs
working on macOS to be able to build (albeit slower than before).

Fixes the following build failure when goma is enabled and running
on macOS Ventura 13.5:

    $ autoninja -C out/Debug-iphonesimulator chrome
    ninja: Entering directory `out/Debug-iphonesimulator'
    [0/3574] CXX ....oninja: fatal: pipe: Too many open files

Bug: 1467777
Change-Id: Ia7eaab552f7e6d26a2f48d72bb8235a70d6d442f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4720227
Reviewed-by: Dirk Pranke <dpranke@google.com>
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
2 years ago
Ben Segall eb2866e654 Create reclient specific ninja wrapper to properly handle reproxy lifecyle
This is required as it is impossible to catch a keyboard interrupt in a
windows batch file and we dont want zombie reproxy instances running on
developer's machines for performance and metric collection reasons.

Bug: b/264405266
Change-Id: I00f036c8f14451cdb1b99a5cad1c2af03dd57d57
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4171506
Reviewed-by: Dirk Pranke <dpranke@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Simeon Anfinrud aec39c38f8 (Reland) Detect RBE builds and accelerate them.
Chromecast builds use a buildflag called `use_rbe` instead of
`use_remoteexec` or `use_goma`.

Previous attempt at crrev.com/c/4144469 tried recycling the
use_remoteexec variable for the use_rbe case, but that caused
problems with Cast CI because it would hit the error case
where the reclient binary isn't found.

This new attempt introduces a new dedicated use_rbe variable
that skips that check intended only for use_remoteexec.

Bug: b/266099996
Test: run `autoninja` in the chromecast internal repo
Change-Id: Ieaf3af709589fe1b8611904afc2fd80284b333b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4179133
Auto-Submit: Simeon Anfinrud <sanfin@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Ben Segall 7954ed74a2 Ensure boostrap shutdown runs if ninja is killed with a signal
This fixes the issue for linux. I will follow up with a windows fix

Bug: 264405266
Change-Id: I518412b06e410d82d02a085180a24987f9ba98fd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4134101
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
2 years ago
Matt Swartwout 762f1e0673 Revert "Detect RBE builds and accelerate them."
This reverts commit f4fa0cc7c1.

Reason for revert: This broke chromecast infra

Original change's description:
> Detect RBE builds and accelerate them.
>
> Chromecast builds use a buildflag called `use_rbe` instead of
> `use_remoteexec`.
>
> Bug: None
> Test: run `autoninja` in the chromecast internal repo
> Change-Id: Iebb18ad47a30c4152923f48175222ec6a20e16f4
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4144469
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Auto-Submit: Simeon Anfinrud <sanfin@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Bug: None
Change-Id: Ib93820b6045088b7dcb45d1f14c0d4f83dc92ebc
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4157771
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Matt Swartwout <mwswartwout@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Simeon Anfinrud f4fa0cc7c1 Detect RBE builds and accelerate them.
Chromecast builds use a buildflag called `use_rbe` instead of
`use_remoteexec`.

Bug: None
Test: run `autoninja` in the chromecast internal repo
Change-Id: Iebb18ad47a30c4152923f48175222ec6a20e16f4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4144469
Reviewed-by: Gavin Mak <gavinmak@google.com>
Auto-Submit: Simeon Anfinrud <sanfin@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Junji Watanabe ad452a7204 Fix autoninja to call ninja.py
ninja.exe will be removed from depot_tools soon.
Calling ninja.py finds the ninja installed by DEPS or one in PATH.

Bug: 1340825
Change-Id: I32ed6d8e5b0498347042490eedeca19972d35233
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4061247
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
2 years ago
Michael Spang f4670898b9 Revert "Update autoninja to check for enable_rbe_bootstrap on builds that use"
This reverts commit f52f44b287.

Reason for revert: Breaks various internal Nest builds

Original change's description:
> Update autoninja to check for enable_rbe_bootstrap on builds that use
> it.
>
> Autoninja at this point still assumes that build directories are two
> levels up from the chromium/src directory.  This will be addressed in a
> follow-on cl.
>
> Bug: b/169983312,b/253452972
> Change-Id: I13eb1e5d8ebfb83d565ea84b6a89c7784eb7fde3
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3975409
> Commit-Queue: Michael Savigny <msavigny@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Bug: b/169983312,b/253452972
Change-Id: Idad4caf46f12860fb4804251f62d0fb498add27d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3982826
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Michael Savigny f52f44b287 Update autoninja to check for enable_rbe_bootstrap on builds that use
it.

Autoninja at this point still assumes that build directories are two
levels up from the chromium/src directory.  This will be addressed in a
follow-on cl.

Bug: b/169983312,b/253452972
Change-Id: I13eb1e5d8ebfb83d565ea84b6a89c7784eb7fde3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3975409
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2 years ago
Michael Savigny 1cf1fb5d21 Add reclient error messaging for developer builds.
At the moment reclient isn't supported for developer builds.  Add
messaging to autoninja to indicate this if a developer attempts a build
with use_remoteexec set.

Bug:b/199192766
Change-Id: I9e016a09c98b756018505f661afdb6eaf3db3243
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3839070
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
3 years ago
Richard Wang bb07d9eb0b Remove use_rbe gn arg support
As it is removed in https://crrev.com/c/3744967, this cleans up the check for a no longer existing arg.

Bug: chromium:1247781
Change-Id: Ic7288cf34e63cd52087e2b51f1b4c816bbbc0363
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3745201
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Richard Wang <richardwa@google.com>
3 years ago
Sylvain Defresne cb2cef9a77 [mac] Restore an upper limit on the number of jobs on macOS
Even when increasing the maximum number of file descriptor
as recommended in the documentation, using a large number
of concurrent jobs cause build error on certain macOS devices
(mostly iMacPro with 18 cores which ends up with a limit of
1440 jobs, and end up reaching the heightened file descriptor
limit).

Put a limit of 800 which has been found to work on all of
the devices available and still allow to have a multiplier
of 80 on highest end M1 devices (currently 10 cores).

Bug: 1317620, 936864
Change-Id: I32560c5ae9462e94f61a773d625ef3758bf05ee5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3634807
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Sylvain Defresne <sdefresne@chromium.org>
Commit-Queue: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
3 years ago
Aleksey Khoroshilov 1bc3cd228e Add a global core limit support for goma/rbe builds.
This is very usable for non Google-sized Goma clusters. We want to use
NINJA_CORE_MULTIPLIER logic, but at the same time we don't want to stall
our cluster with super-high -j values.

Bug: 1323475
Change-Id: I2ed414463b4f397a7bcebe42ea0b996e52006cda
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3632492
Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Takuto Ikuta 6a1494e5d7 Reland "autoninja: increase parallelism for non-HT cpu"
This reverts commit 564f1eee96.

Reason for revert:
fixed platform import

Original change's description:
> Revert "autoninja: increase parallelism for non-HT cpu"
>
> This reverts commit 06b18e4b69.
>
> Reason for revert:
> platform is not imported
>
> Original change's description:
> > autoninja: increase parallelism for non-HT cpu
> >
> > M1 mac seems to have capacity to build with higher parallelism.
> > This also remove limit in macOS.
> >
> > Bug: 1317620
> > Change-Id: I4460915c405cbb27ed977dcee631adb8753f2335
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3596361
> > Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> > Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> > Reviewed-by: Junji Watanabe <jwata@google.com>
> > Reviewed-by: Fumitoshi Ukai <ukai@google.com>
> > Reviewed-by: Philipp Wollermann <philwo@google.com>
> > Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
>
> Bug: 1317620
> Change-Id: I325d0be8feca28e2f89a9191cfdb01de1f367d86
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3607513
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> Owners-Override: Takuto Ikuta <tikuta@chromium.org>

Bug: 1317620
Change-Id: I2c3d817ca25cf6654864823788d3a8c473168942
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3612482
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
3 years ago
Takuto Ikuta 381db68adc autoninja: add simple test
This is to prevent revert like https://crrev.com/c/3607513

Bug: 1317620
Change-Id: I6ab7aba5f92719bd573d22d90358f58e48aeb10c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3607514
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
3 years ago
Takuto Ikuta 564f1eee96 Revert "autoninja: increase parallelism for non-HT cpu"
This reverts commit 06b18e4b69.

Reason for revert:
platform is not imported

Original change's description:
> autoninja: increase parallelism for non-HT cpu
>
> M1 mac seems to have capacity to build with higher parallelism.
> This also remove limit in macOS.
>
> Bug: 1317620
> Change-Id: I4460915c405cbb27ed977dcee631adb8753f2335
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3596361
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: Fumitoshi Ukai <ukai@google.com>
> Reviewed-by: Philipp Wollermann <philwo@google.com>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Bug: 1317620
Change-Id: I325d0be8feca28e2f89a9191cfdb01de1f367d86
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3607513
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: Takuto Ikuta <tikuta@chromium.org>
3 years ago
Takuto Ikuta 06b18e4b69 autoninja: increase parallelism for non-HT cpu
M1 mac seems to have capacity to build with higher parallelism.
This also remove limit in macOS.

Bug: 1317620
Change-Id: I4460915c405cbb27ed977dcee631adb8753f2335
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3596361
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Aravind Vasudevan c5f0cbb865 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 42c5bbbc96 Revert "Use pylint 2.7 for depot_tools"
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan 22bf605bb6 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Takuto Ikuta 237cc469e4 switch autoninja shebang to python3
We don't want to use python2 and this doesn't use third party library
too so no need to be vpython which has a small overhead.

Change-Id: Iff924a87f0562069e29195a5693e2260e4a6684e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400397
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago