Account check would become too slow.
We'll check RBE project to use instead.
On corp machine, our policy to use @google.com account
and rbe-chrome-untrusted to build chromium/chrome.
We don't allow rbe-chromium-untrusted with @chromium.org
on corp machine.
On non-corp machine, user couldn't use rbe-chrome-untrusted
because it's @google.com only, and corp security policy
doesn't allow @google.com account on non-corp machine.
Bug: b/364318216
Change-Id: I0f3a19e105b050aef6a62e1b25b45b1722382a34
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5848450
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Currently, {autoninja, autoninja.bat} scripts generate AUTONINJA_BUILD_ID by calling `python3 -c "import uuid; print(uuid.uuid4())"`.
This CL moves the logic at the beginning `autoninja.py`.
This also reduces the overhead of starting Python interpreter, which
takes around 60ms on my Windows workstaion.
```
> hyperfine python3
Benchmark 1: python3
Time (mean ± σ): 63.4 ms ± 7.8 ms [User: 25.7 ms, System: 33.7 ms]
Range (min … max): 51.0 ms … 80.2 ms 41 runs
```
Change-Id: I93d5b65f3c5542c3a93a3de1f27a5aa5d06c09a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5737673
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.
The user consent message will be displayed only once at the beginning of a build.
```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***
Proxy started successfully.
...
```
Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This disallows Googler to use non-google account with reclient and
siso from corp machine.
Bug: b/309720176
Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells.
Part of my bigger auto{ninja,siso} refactoring.
Tested:
- Handling of the ^^ suffix on Windows still works correctly.
- Handling of error codes - i.e.; making sure
"autoninja base_unittests && base_unittests.exe" behaves properly
in the success/failure case.
- Make sure the command prompt title is reliably reset on exit.
I tested autoninja with all combinations of these:
- Host platform: Linux, macOS, Windows
- Remote GN args: <none>, use_goma=true, use_remoteexec=true
- Siso GN args: <none>, use_siso=true
- Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows)
R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org
Bug: b/293657720
Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
The version of expr shipped on macOS does not support `substr`
command and prints `expr: syntax error`. Change the invocation
of `expr` to use the `:` operator which match rhs regexp into
lhs string and return the length of the match.
This should be supported by all versions of `expr` that are
POSIX compliant.
Bug: 1199260
Change-Id: Ib932702fedbbbc772d65086ebc371dc0a9023c0e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2827931
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
We switched autoninja.py to python3 in https://crrev.com/c/2752772
but that assumes `python3` is in $PATH (which is likely true on
Mac and Linux but less likely to be true on Windows, especially
in MSYS/MinGW/git bash shells.
It's safer to use the `python3` bundled into depot_tools in
`$depot_tools/python-bin` and more hermetic to boot.
Bug: 1187609
Change-Id: I65efd1a2b87135ac4e25ba377309c3374cfda085
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2756889
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit 428143ee24.
Reason for revert: Fixing the issues revealed by the original change by
avoiding python3 and by checking for the existence of gomacc[.exe]
before running it.
This also relands 2241db8a1f - "Avoid
capture_output to support Python 3.6", to simplify relanding and any
possible reverts.
Original change's description:
> Revert "Check whether goma is running when it is enabled"
>
> This reverts commit b7ddc5a009.
>
> Reason for revert:
> This broke the builder where depot_tools is not in PATH.
> https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8858077852309878080/+/u/build/stdout
>
> Original change's description:
> > Check whether goma is running when it is enabled
> >
> > One of the mistakes one can make when running ninja is having goma
> > enabled (use_goma=true in args.gn) but not having goma running. This can
> > lead to ~1,000 failed compile steps, which is messy.
> >
> > This change teaches autoninja.py to check whether goma is running. If
> > not then it tells autoninja to just print a warning message. The
> > check costs roughly 30 ms which seems reasonable.
> >
> > In fact, because this change also switches away from vpython (necessary
> > to use python3 to use subprocess.run) it actually runs about 600 ms
> > _faster_ than before this change.
> >
> > If build acceleration is requested through use_rbe then no checking for
> > whether the service is running is done. That could be added in the
> > future.
> >
> > autoninja.py could auto-start goma but that is error prone and has
> > limited additional value.
> >
> > This was tested on Linux, OSX, and Windows.
> >
> > Bug: 868590, b/174673874
> > Change-Id: Ie773e574878471e5136b9b82d52f86af3d848318
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2627014
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> > Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
>
> TBR=yyanagisawa@google.com,dpranke@google.com,brucedawson@chromium.org,sanfin@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
>
> Change-Id: I57a6c73ea853259f3d1ec7ad0ce51e495acc96db
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug: 868590
> Bug: b/174673874
> Bug: 1167064
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2632018
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
> Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@google.com>
TBR=yyanagisawa@google.com,dpranke@google.com,brucedawson@chromium.org,sanfin@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
# Not skipping CQ checks because this is a reland.
Bug: 868590
Bug: b/174673874
Bug: 1167064
Change-Id: I8aa6830259bc18f8e7926cd0bf5c62e671c74a2d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2634201
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This reverts commit b7ddc5a009.
Reason for revert:
This broke the builder where depot_tools is not in PATH.
https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8858077852309878080/+/u/build/stdout
Original change's description:
> Check whether goma is running when it is enabled
>
> One of the mistakes one can make when running ninja is having goma
> enabled (use_goma=true in args.gn) but not having goma running. This can
> lead to ~1,000 failed compile steps, which is messy.
>
> This change teaches autoninja.py to check whether goma is running. If
> not then it tells autoninja to just print a warning message. The
> check costs roughly 30 ms which seems reasonable.
>
> In fact, because this change also switches away from vpython (necessary
> to use python3 to use subprocess.run) it actually runs about 600 ms
> _faster_ than before this change.
>
> If build acceleration is requested through use_rbe then no checking for
> whether the service is running is done. That could be added in the
> future.
>
> autoninja.py could auto-start goma but that is error prone and has
> limited additional value.
>
> This was tested on Linux, OSX, and Windows.
>
> Bug: 868590, b/174673874
> Change-Id: Ie773e574878471e5136b9b82d52f86af3d848318
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2627014
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
TBR=yyanagisawa@google.com,dpranke@google.com,brucedawson@chromium.org,sanfin@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
Change-Id: I57a6c73ea853259f3d1ec7ad0ce51e495acc96db
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 868590
Bug: b/174673874
Bug: 1167064
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2632018
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@google.com>
One of the mistakes one can make when running ninja is having goma
enabled (use_goma=true in args.gn) but not having goma running. This can
lead to ~1,000 failed compile steps, which is messy.
This change teaches autoninja.py to check whether goma is running. If
not then it tells autoninja to just print a warning message. The
check costs roughly 30 ms which seems reasonable.
In fact, because this change also switches away from vpython (necessary
to use python3 to use subprocess.run) it actually runs about 600 ms
_faster_ than before this change.
If build acceleration is requested through use_rbe then no checking for
whether the service is running is done. That could be added in the
future.
autoninja.py could auto-start goma but that is error prone and has
limited additional value.
This was tested on Linux, OSX, and Windows.
Bug: 868590, b/174673874
Change-Id: Ie773e574878471e5136b9b82d52f86af3d848318
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2627014
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
Also make autoninja.py a vpython script to use reliable version of
psutil.
Note: this change also makes autoninja always make a decision about
-j; there's no longer a default where it lets ninja pick. The code is
simpler this way and I think it's better because it lets developers
always see which -j is in effect when using autoninja (and that's its
exact purpose, if you wanted default you shouldn't have used autoninja).
R=dpranke@chromium.org
Bug: 976265
Change-Id: Ic9d12469729e4bf58da1ec1bd70437329519fc46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1663904
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Gabriel Charette <gab@chromium.org>
Auto-Submit: Gabriel Charette <gab@chromium.org>
I want to use autoninja from //build/android/generate_gradle.py, but I
invoke it with a *long* list of targets that I do not want echo'ed
Bug: 803493
Change-Id: I98ad85b64ec7ce60b7e3df9827fb613cd57638e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1596535
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
I noticed current autoninja does not upload build log in failed build.
Change-Id: Ie58646b483e130769ad22113953c0b95b3145548
Reviewed-on: https://chromium-review.googlesource.com/c/1441892
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
This CL introduces ninjalog_uploader to autoninja for posix environment.
This is to collect build statistics from googler to find user side build performance bottleneck.
ninjalog_uploader_wrapper is used to notify user and manage small config of upload script.
Bug: 900161
Change-Id: I48ac8cd1d52f64e8fdafaec43636d2d79ef9040d
Reviewed-on: https://chromium-review.googlesource.com/c/1345255
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
On Windows, without quote, whole return value would be treated as
a command, and the execution would fail.
On Linux and Mac, without quote, if depot_tools is settled under
directories with ' ', command execution would fail because paths are
separated in a wrong way.
To make such a return value work on Linux and Mac, the shell script
started to use eval.
Bug: 902930
Change-Id: I9bb74585294af565988c0b844b6b113a5c685530
Reviewed-on: https://chromium-review.googlesource.com/c/1325249
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
For ease of distinguishing a series of build from others in Goma,
let me make autoninja set unique build ID.
The ID can be usable for statistics purpose to see how much time is
usually spent for each build.
Since the unique ID is not specific to Goma, i.e. anybody invoked by
ninja can use it to distinguish builds, I chose AUTONINJA_BUILD_ID.
Since we do not need to distinguish each node, I chose uuid4.
Bug: b/77176746
Change-Id: I0b6a67811b420c51d1fd3fd113bf7f039a41e2ab
Reviewed-on: https://chromium-review.googlesource.com/c/1275685
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
A handy usage pattern for autoninja.bat that I was not aware of is to go
autoninja -C out\Default chrome && chrome
This will build chrome and then run it, but only run it if the build
succeeds. The addition of post_build_ninja_summary.py broke this by
losing the error code. This change fixes it by using black magic to
set an error code in the failure case.
Bug: chromium:787983
Change-Id: Ib87fd1799816e19d56de76e08e0f9688be903d80
Reviewed-on: https://chromium-review.googlesource.com/916705
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
post_build_ninja_summary.py gives a summary of a ninja build. It can be
run standalone or it can be run automatically by autoninja. This CL
updates the Python script and the autoninja bash script to make this
work on Linux. This includes removing a zero-value assert, and ensuring
that .so files get categorized as such.
Change-Id: I2d59ab129f5ce70117beeb119719f8432bfbab7c
Reviewed-on: https://chromium-review.googlesource.com/915053
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Using goma requires the developer to remember which build directories
use goma and which don't so that they can pass an appropriate -j number.
Getting this wrong makes builds slower, either by under utilizing
resources or by causing a self-inflicted DOS attack. Usage:
autoninja -C out/debug
autoninja looks at the settings for the specified build directory and
then selects either -j num_cores*20 or no -j flag based on the
use_goma setting.
You can set the NINJA_CORE_MULTIPLIER variable to change from the
default 20* multiplier. You can also use NINJA_CORE_ADDITION if you
want non-goma builds to specify -j with an offset to the number of
cores, such as this Linux command:
NINJA_CORE_ADDITION=-2 autoninja -C out/release base
This will tell autoninja to pass -j to ninja with num_cores-2 as the
parameter.
On Windows you can have a ninja.bat file (ahead of ninja on the path)
such that autoninja will automatically be used. It should contain this:
@call autoninja.bat %*
Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655
Reviewed-on: https://chromium-review.googlesource.com/517662
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Fumitoshi Ukai
Junji Watanabe
Takuto Ikuta
Philipp Wollermann
Yannic Bonenberger
Sylvain Defresne
Jamie Madill
Dirk Pranke
Bruce Dawson
Yoshisato Yanagisawa
Peter Kasting
Edward Lemur
Marc-Antoine Ruel
Gabriel Charette
Andrew Grieve
Yoshisato Yanagisawa