Commit Graph

108 Commits (d1259b49c5b41bc4939ba7c5bb8e8e08083a05f6)

Author SHA1 Message Date
Mark Mentovai 22df6f8e62 autoninja: fix strings with invalid escape sequences
Since 61fad561d6 (https://chromium-review.googlesource.com/c/5848450,
2024-09-11), autoninja under Python 3.12 presents these warnings:

```
…/autoninja.py:73: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('instance\s*=\s*projects/([^/]*)/instances/.*', line)
…/autoninja.py:92: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('SISO_PROJECT=\s*(\S*)\s*', line)
```

This warning appears in Python 3.12 ([1], [2], [3]).

'\s' and '\S' are not valid escape sequences in strings. r'\s' and r'\S'
are valid in regular expressions, but outside of raw strings, they would
need to be written as '\\s' and '\\S'.

There is no reason to not use raw strings in this case, so the new
regular expression pattern strings introduced in 61fad561d6 are
changed to raw strings.

[1] https://docs.python.org/3/whatsnew/3.12.html#:~:text=A%20backslash%2Dcharacter%20pair%20that%20is%20not%20a%20valid%20escape%20sequence%20now%20generates%20a%20SyntaxWarning%2C%20instead%20of%20DeprecationWarning.
[2] https://github.com/python/cpython/issues/98401
[3] https://github.com/python/cpython/pull/99011

Bug: b/364318216
Change-Id: I0f237976fe9c39208541ae78205f5bdbf126fa82
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5859159
Commit-Queue: Mark Mentovai <mark@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Auto-Submit: Mark Mentovai <mark@chromium.org>
7 months ago
Max Ihlenfeldt 62475a5ed8 autoninja.py: Fix "SyntaxWarning: invalid escape sequence '\s'"
Bug: None
Change-Id: Id19557ff5f115f1cacee85f113174535112f7c9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5854399
Commit-Queue: Philipp Wollermann <philwo@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Auto-Submit: Max Ihlenfeldt <max@igalia.com>
8 months ago
Aleksey Khoroshilov 1ad5b6c0df Do not fail if RBE project is empty.
External developers may use RBE without RBE_instance set.
Autoninja should not fail in this case.

Change-Id: I2dbafded690246132bb0ff1cd080caa0b758b4f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5851434
Auto-Submit: Aleksei Khoroshilov <akhoroshilov@brave.com>
Commit-Queue: Aleksei Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
8 months ago
Fumitoshi Ukai 61fad561d6 autoninja: check RBE project, not account
Account check would become too slow.
We'll check RBE project to use instead.

On corp machine, our policy to use @google.com account
and rbe-chrome-untrusted to build chromium/chrome.
We don't allow rbe-chromium-untrusted with @chromium.org
on corp machine.

On non-corp machine, user couldn't use rbe-chrome-untrusted
because it's @google.com only, and corp security policy
doesn't allow @google.com account on non-corp machine.

Bug: b/364318216
Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
8 months ago
Fumitoshi Ukai 1f6ef165b7 autoninja: show warning if siso builtin remote exec is not used on cog
If user forgets to set `use_reclient=false`,

```
ukai@ukai /google/cog/cloud/ukai/siso-test/chromium
 % ~/depot_tools/autoninja -C out/Default base
WARNING: You're not using Siso's built-in remote execution. The build will be slow.
You should set the following in args.gn to get better performance:
  use_remoteexec=true
  use_reclient=false
  use_siso=true

Proxy started successfully.
```

Bug: b/362301140
Change-Id: I40972b96fd2576c32232f88e6248a1000708c3e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5816108
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Gary Tong <gatong@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@google.com>
8 months ago
Junji Watanabe 8f617b4617 autoninja: Add exit code and build duration to ninjalog metadata
- exit code will be useful to distinguish between successful builds and
others.
- build duration will cover the overhead outside of ninja invocation
e.g. Reclient and Siso's startup/shutdown.

Bug: 348527311
Change-Id: Ibebdf3d64597fb0e57914a84d33e46bca66671da
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5804201
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
8 months ago
Junji Watanabe 1e6454f8d2 autoninja: ignore ResourceWarning from subprocess for ninjalog_uploader.py
The main process exits right after ninja_uploader.py subprocess starts,
which may print ResourceWarning from subprocess.
e319f774f9/Lib/subprocess.py (L946-L947)
This CL ignores the warning message.

Change-Id: I800387194763e60a16437b3bd41fa282b605dda7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5782795
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
9 months ago
Andrew Grieve 5f6542c176 autoninja.py: Detect when use_siso=false exists after use_siso=true
I ran into this from having use_siso=true in an include(), and then
trying to turn it off again via an assignment

Change-Id: Iba70ade68c7a3699755d6a0070816283e0538c6f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5769918
Commit-Queue: Junji Watanabe <jwata@google.com>
Auto-Submit: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
9 months ago
Junji Watanabe 670ce748be autoninja: Propgate AUTONINJA_BUILD_ID to siso ninja via SISO_BUILD_ID
Bug: 355127782
Change-Id: Ia7f55696e9282c8719ca98892963bb991caa70f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5734819
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
9 months ago
Junji Watanabe 8552a27900 autoninja: Generate UUID for AUTONINJA_BUILD_ID inside autoninja.py
Currently, {autoninja, autoninja.bat} scripts generate AUTONINJA_BUILD_ID by calling `python3 -c "import uuid; print(uuid.uuid4())"`.
This CL moves the logic at the beginning `autoninja.py`.

This also reduces the overhead of starting Python interpreter, which
takes around 60ms on my Windows workstaion.

```
> hyperfine python3
Benchmark 1: python3
  Time (mean ± σ):      63.4 ms ±   7.8 ms    [User: 25.7 ms, System: 33.7 ms]
  Range (min … max):    51.0 ms …  80.2 ms    41 runs
```

Change-Id: I93d5b65f3c5542c3a93a3de1f27a5aa5d06c09a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5737673
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
9 months ago
Junji Watanabe 1a616deaac autoninja: Split args before passing it to _upload_ninjalog()
autoninja passes args as a concatenated string.
It needs to be split before _upload_ninjalog().

Bug: 345113094
Change-Id: Ia993233dca9c6a0c2be1cc9080ab063a6db3f8d8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5724920
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
9 months ago
Junji Watanabe cdcdd6efee ninjalog: Add --cmdline ninjalog_uploader.py option
https://crrev.com/c/5669094 dropped `--cmdline` by mistake.

Bug: 345113094
Change-Id: Ia282a4a18195e000ae658ebc96467f797657f4b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5724910
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
9 months ago
Junji Watanabe b12257963c autoninja: Replace ninjalog_uploader_wrapper.py and reclient_metrics.py with build_telemetry.py
This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.

The user consent message will be displayed only once at the beginning of a build.

```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***

Proxy started successfully.
...
```

Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
9 months ago
Junji Watanabe 1387a8c958 autoninja: Refactor the code calling Reclient
- Remove reclient_helper.py to avoid calling it directly.
- Cleans up autoninja.py

I just want to do this cleanup before merging the user notice messages
for logs/metrics collection.

Bug: 345113094
Change-Id: I0c76426b3cb387eae6dede031727c107e57d5d1a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5668282
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
10 months ago
Fumitoshi Ukai 356ef0324e ninja: error if trying to build for use_remoteexec=true
Change-Id: Ia32dd3cba1d1874401c6614f792f212b2cfa60a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5660200
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
10 months ago
Fumitoshi Ukai e6d8f67fae autoninja: set PYTHONDONTWRITEBYTECODE=1 if not set
during build, lots of python scripts run and generates
lots of pycache *pyc in workspace.
To keep workspace clean, set PYTHONDONTWRITEBYTECODE=1
in autoninja.py

Change-Id: I5ce4c904aa11dd8172672ff9a8e9a06ebf69c9fe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5599922
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gary Tong <gatong@chromium.org>
Reviewed-by: Gregory Guterman <guterman@google.com>
11 months ago
Fumitoshi Ukai da5e2f948a autoninja: fix for no args
need to split input_args[1] even if it doesn't contains " ",
e.g. when input_args[1] == ""

Bug: b/343172216
Change-Id: I32d9346063534ea2b4f891fd596c7ec02ab10dbf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5583415
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Richard Wang <richardwa@google.com>
Reviewed-by: Richard Wang <richardwa@google.com>
11 months ago
Junji Watanabe 8a11c2d152 autoninja: Support Siso builtin RBE client mode
This CL allows autoninja to use Siso's builtin RBE client instead of using Reclient.
Developers have to specify `use_remoteexec=true use_reclient=false` GN arg to use this mode.

This should be no-op for all developers unless `use_reclient=false` is set.

Bug: 342038479
Change-Id: I8a67f114c056be6ddb0cd26cccf0157e0b184c5a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5567811
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
11 months ago
Junji Watanabe 450c36420d autoninja: Remove autosiso
autosiso has been deprecated. Nobody should be using it now.

Bug: 342038479
Change-Id: I3b22b9fc6db5b4693e954818c5aed58a3e4b4a0d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5567813
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Junji Watanabe <jwata@google.com>
11 months ago
Takuto Ikuta 401216ed9b autoninja: fix behavior when passed args after '--'
This is to make command like below works with autoninja.
```
$ autoninja -C out/Default -- base
```

This '--' is inserted in
https://source.chromium.org/chromium/chromium/tools/build/+/main:recipes/recipe_modules/tricium_clang_tidy/resources/tricium_clang_tidy_script.py;l=129;drc=c2f28b4ccdaad78041caf6710edc2f75f44160ae

Bug: 336474469
Change-Id: Iad326ed592d7c968073fd6f4a89c06763992c51f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5472533
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Takuto Ikuta 6aaae85821 autoninja: remove goma references from autoninja
Bug: b/304421889
Change-Id: I2f77f4d8e049ad7ae500e995a1ae2444688b96da
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5456272
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Michael Savigny 8de9817392 Add luci-auth account check for non-google on corp.
Include a luci-auth check of the account logged in.

This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.

Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
1 year ago
L. David Baron 79e1a7174c autoninja: Show additional help information for --help, like for -h.
This doesn't catch all the variants of --help that getopt allows but it
does catch --help, which is probably the most common.

Bug: 329317224
Change-Id: I2dde63df84bd2c2ade5f9d2c1ab10c3dc13fba70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370829
Auto-Submit: David Baron <dbaron@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
1 year ago
Michael Savigny beb48f193f Minor change to -h output for autoninja.
Notes that --offline is for disabling remote execution in general not just for goma.

Bug: b/329317224
Change-Id: I5538c9fac6d5161e98e1d156e61c512c6a1e8496
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370094
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
1 year ago
Michael Savigny 50de666ba4 Revert "Remove ability to use_goma in autoninja (chromeos excepted)"
This reverts commit 6b84fbfb20.

Reason for revert: Jumped the gun before final resolution for b/316921158

Original change's description:
> Remove ability to use_goma in autoninja (chromeos excepted)
>
> Won't run a build if use_goma=true and target_os!="chromeos".
>
> Bug: b/277197166
> Test: Manual tests and updates to unit tests.
> Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>

Bug: b/277197166
Change-Id: Iaf506a10dfdc1c1672b2927636adf5773edc67c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333176
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
1 year ago
Michael Savigny 6b84fbfb20 Remove ability to use_goma in autoninja (chromeos excepted)
Won't run a build if use_goma=true and target_os!="chromeos".

Bug: b/277197166
Test: Manual tests and updates to unit tests.
Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta 280bb93210 Revert "Remove ability to use_goma via autoninja."
This reverts commit 1592a89c9f.

Reason for revert:
b/325856422

Original change's description:
> Remove ability to use_goma via autoninja.
>
> Doesn't run a build if use_goma is true, point to reclient docs.  Turns
> off use_goma for linux and mac builds.  Windows will follow in 2-3 weeks.
>
> Bug: b/277197166
> Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
> Reviewed-by: Ben Segall <bentekkie@google.com>
> Reviewed-by: Ramy Medhat <abdelaal@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>

Bug: b/277197166 b/325856422
Change-Id: Iaec83b254af002f420bfd2df1154ce94e18fac15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5308674
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: David Munro <davidmunro@google.com>
Reviewed-by: David Munro <davidmunro@google.com>
1 year ago
Michael Savigny 1592a89c9f Remove ability to use_goma via autoninja.
Doesn't run a build if use_goma is true, point to reclient docs.  Turns
off use_goma for linux and mac builds.  Windows will follow in 2-3 weeks.

Bug: b/277197166
Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
Reviewed-by: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 26b100def9 Update goma deprecation message and add delay to build start for goma.
Update the warning message for linux and mac goma builds, and add a 5
second delay.

Do not submit until Jan 22, 2024

Bug: b/277197166
Change-Id: Id93822b1aa32b05746b8fad4b9dc332cf61d291c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5215930
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta 3a7f62489a autoninja: make it possible to skip external account check
This is for some internal infrastructures which want to use service
accounts from corp machines.

Bug: b/320639792
Change-Id: Ie90241fe89e0bb0fb5526844e2e3651cc316d945
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5226384
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Richard Wang <richardwa@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Michael Savigny 83458774b1 Update autoninja use_goma warning message.
Update use_goma warning for linux and mac, to add date of removal of
use_goma (Feb 7th) and a warning that delays will be added (Jan 22).

Bug: b/277197166
Change-Id: Id567e5bf704c5d80e5122663c6d8194ac510d992
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5200850
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Michael Savigny f26dddb204 Add use_goma deprecation warning to autoninja.
Adds the same warning to every use_goma build that is currently printed
on a gn gen with use_goma=true.

Bug:b/277197166
Change-Id: Icf9781f814416dfbeb2826fc88372465155fcd2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5123231
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta aa65be9856 autoninja: cache negative result of gcloud account check
This is to remove 1~3 seconds from no-op build using autoninja.

Bug: b/309720176
Change-Id: I2c15a43517d5a99eae794015c18c884a3f3fdef4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5092847
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
1 year ago
Takuto Ikuta cccca95d8f Revert^2 "autoninja.py: disallow external account from corp machine"
This reverts commit ed596dadf9.

Reason for revert:
Ignore exception when getting account for application default
credentials.

Original change's description:
> Revert "autoninja.py: disallow external account from corp machine"
>
> This reverts commit 0d0f28a4fc.
>
> Reason for revert:
> http://b/309720176#comment7
>
> Original change's description:
> > autoninja.py: disallow external account from corp machine
> >
> > This disallows Googler to use non-google account with reclient and
> > siso from corp machine.
> >
> > Bug: b/309720176
> > Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
> > Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> > Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
>
> Bug: b/309720176
> Change-Id: I62b4891b1e2e2554d088d162057982b95c8adcd7
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065849
> Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> Owners-Override: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
> Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Bug: b/309720176
Change-Id: Ib9aef1b7fa8e3c9bb20ac8d6b461c6bcff72dc83
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065052
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Takuto Ikuta ed596dadf9 Revert "autoninja.py: disallow external account from corp machine"
This reverts commit 0d0f28a4fc.

Reason for revert:
http://b/309720176#comment7

Original change's description:
> autoninja.py: disallow external account from corp machine
>
> This disallows Googler to use non-google account with reclient and
> siso from corp machine.
>
> Bug: b/309720176
> Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
> Reviewed-by: Philipp Wollermann <philwo@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Takuto Ikuta <tikuta@chromium.org>

Bug: b/309720176
Change-Id: I62b4891b1e2e2554d088d162057982b95c8adcd7
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5065849
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Owners-Override: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
1 year ago
Takuto Ikuta 0d0f28a4fc autoninja.py: disallow external account from corp machine
This disallows Googler to use non-google account with reclient and
siso from corp machine.

Bug: b/309720176
Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Takuto Ikuta df3e577855 apply black and `git cl format` for ninja related python files
This is made by
$ black --line-length 79 *ninja*.py
$ git cl format

Change-Id: Ic446898a5461ae536542f6312cae2ce126dfe82a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035265
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
1 year ago
Takuto Ikuta 5cbc521beb autoninja.py: remove use_rbe support
This arg was not used anymore.

Change-Id: If578845fc3e200f4f23c8ca050542b92fcf13ef2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035261
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
1 year ago
Philipp Wollermann 17debef507 autoninja.py: Enable offline mode for Siso when use_remoteexec not set
Tested with:

$ autoninja -C out/fastbuild-siso-local base
offline mode
 2.48s loading build.ninja
pre:0 local:176 remote:0 5.7/s fallback:0
[...]
5m40.86s Build Succeeded: 1641 steps - 4.81/s

$ autoninja -C out/fastbuild-siso-reclient base
Running gcloud auth print-access-token --trace-token=interactiveauth
Proxy started successfully.
124.260 s to start reproxy
 5.32s loading build.ninja
15.93s Regenerating ninja files
 2.75s update filegroups
 4.84s loading build.ninja
pre:0 local:1 remote:0 30.2/s cache:100.00% fallback:0
[...]
2m05.55s Build Succeeded: 2560 steps - 20.39/s

R=jwata, ukai

Change-Id: Ib976b874796a20dca55005986ee6f7ee3c942245
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4997466
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
2 years ago
Fumitoshi Ukai 406be8281e autoninja: don't allow use_siso=true and use_goma=true
Bug: b/306048338
Change-Id: Ib34eb5abe3c419d4afefb7a9cd083441ef755110
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4951553
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
2 years ago
Junji Watanabe c51829968b [autoninja] Update Siso .ninja_log comments
Siso is going to generate non-empty .ninja_log for the analysis tools.
https://crrev.com/c/4907677

This CL fixes the comment about empty ninja_log.

It's not compatible with Ninja's .ninja_log. So `gn clean` will still
be required.

Bug: b/298594790
Change-Id: Ib4c60f3ed22f516d6f7e2847aaf57e228121eccf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4937692
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
2 years ago
Philipp Wollermann 0b943400a4 Execute Ninja / Siso directly from autoninja.py.
Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells.

Part of my bigger auto{ninja,siso} refactoring.

Tested:
- Handling of the ^^ suffix on Windows still works correctly.
- Handling of error codes - i.e.; making sure
  "autoninja base_unittests && base_unittests.exe" behaves properly
  in the success/failure case.
- Make sure the command prompt title is reliably reset on exit.

I tested autoninja with all combinations of these:
- Host platform: Linux, macOS, Windows
- Remote GN args: <none>, use_goma=true, use_remoteexec=true
- Siso GN args: <none>, use_siso=true
- Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows)

R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org

Bug: b/293657720
Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
2 years ago
Henrique Ferreiro 7eb4e4841f Don't limit ninja -j on Linux beyond ulimit -n
macOS limits the ninja j value to 1000, because ninja has a limit to the
number of open file descriptors of FD_SETSIZE, which is 1024 on Darwin.
On Linux, the ninja binary distributed on Chromium seems to be compiled
with poll.h support, so that this limitation doesn't exist:
22b778ca19/src/subprocess-posix.cc (L59)

Change-Id: I97848bb99c08fe118dbdaea525da713382373c9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4866223
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
2 years ago
Samuel Attard 8a25998dd9 handle absolute and relative gn imports in autoninja
Follow up to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864898. Now
that autoninja attempts to parse and follow import directives we
also need to support relative and absolute paths not just project
(//*) references.  This adds supports for those imports in
the added autoninja logic.

Change-Id: I93209b3db76e4e3d296616a3aace32e4d5fc5109
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4869108
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Samuel Attard <samuel.r.attard@gmail.com>
2 years ago
Bruce Dawson 30c1cba02f Handle import statements in args.gn files
Did you know that args.gn files can have import statements and
conditionals? I did not, but apparently some developers make use of both
of these.

Supporting import statements is not too hard, so this change adds this
support. Supporting conditionals is possible, but risks turning
autoninja into a turing complete language which is more than I think we
want to do.

This doesn't use the similar code in tools/mb/mb.py because that
code is complex, and relies on the script location to find the src
directory.

This change also updates two of the existing test conditionals that
were not quite sufficient - ninja/autoninja default to num-cores
plus 2 so > cpu_count() is actually not sufficient to prove
anything.

Bug: 1482404
Change-Id: I0539d8068af59d11927cbdad260278a24ab912e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864898
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Henrique Ferreiro 8bde16478f Improve autoninja resource limit handling
This CL makes autoninja raise the number of allowed open file
descriptors to its hard limit, instead of a random value.

Also, independently of whether this operation was successful or not, the
ninja j value is capped at 80% of the current value of allowed open file
descriptors. This approximately matches the previous value of 200 for
the default value of 256 in macOS, but it's also more future-proof.

Lastly, this changes are also applied on Linux.

Change-Id: Idf2cd08384fe9f2bc699293d7062122590284dba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4852717
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Bruce Dawson 9c4fbc5a2a Improve ninja-to-siso switch detection
autoninja tries to detect if a user switches an output directory from
ninja to siso without doing a "gn clean" between. Initiallly this
detection looked for .ninja_deps, but this file doesn't always exist
after a ninja build. This change switches to looking for .ninja_log.

However autosiso creates a .ninja_log file (currently zero length) so
the new check is:

a) If .siso_deps exists then a siso build has been done and only siso
builds are allowed.
b) If .siso_deps doesn't exist but .ninja_log does then a ninja build
has been done and only ninja builds are allowed.
c) If neither file exists then the directory was cleaned and any type of
build is allowed.

Bug: b/293657720
Change-Id: I2cdba74f0894e62aa0e68f91e225f497b2425f45
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4784103
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
2 years ago
Ben Segall 467991e727 reclient: support -o and --offline flags in autoninja
This ensures that for ninja+reclient builds we still start reproxy to allow `rewrapper clang ...` to run locally.

Also this fixes a preexisting bug where RBE_remote_disabled=1 did not apply to the ninja call when running on mac

Bug: b/295192141
Change-Id: I37c4ffdd5f30d423b84f6c469f5edfa88efe1344
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4659225
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Bruce Dawson 5a4c350ee9 Teach autoninja to handle siso/autosiso
autoninja is intended to mean "build this directory, don't bother me
with the details". As we transition to siso it is therefore appropriate
that it should take on the duty of deciding whether to invoke ninja or
siso.

By looking for a use_siso gn arg autoninja can easily do this.

This change relies on crrev.com/c/4753433 to add the use_siso gn arg.

This change also teaches autoninja to detect if a user switches
between siso and ninja without doing a gn clean inbetween.

Note that this change also teaches autoninja to invoke autosiso or
siso ninja based on whether use_remoteexec is true.

Bug: b/293657720
Change-Id: I3ad36a81857e75ffe6babc4f107b777e733a285b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4749722
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago