On the wasm team we build several CMake-based projects with ninja
(LLVM, binaryen, etc), and its useful for us (me at least) to be able
to use autoninja with these project too.
Change-Id: I7e213448dbbe95ffe3d249c9c6a3d4baa41f50d1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1682742
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Sam Clegg <sbc@chromium.org>
This incorporates a fix to psutil's calculation of
the number of processors on Windows systems with
more than 64 cores.
Windows systems with more than 64 logical
processors divide the processors into groups, with
most applications using only one group, i.e. a
portion of the actual processing power available.
Prior to v5.4.4, psutil only counted the number of
processors in the first processor group. This
resulted in only partial utilization of high-
processing-power systems.
For example, on the 72-core P920, this change has
the following effect:
Before:
> autoninja -C out\nogoma chrome
"c:\src\depot_tools\ninja.exe" -C out\nogoma base -j 38
After:
> autoninja -C out\nogoma chrome
"c:\src\depot_tools\ninja.exe" -C out\nogoma base -j 74
Using this new version of psutil doubles the
number of processors used for building Chrome on
the P920 (when not using goma).
A similar bug exists in ninja.exe, so using
autoninja will be particularly important after
this fix for users of the P920 and other systems
with >64 cores. More fixes will be needed
elsewhere for similar bugs - see crbug.com/980967
for general progress.
Bug: 980270
Change-Id: I8de61a72cf95acf28ef1bcef1b0057b7b1225832
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1686081
Commit-Queue: Jesse McKenna <jessemckenna@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Also make autoninja.py a vpython script to use reliable version of
psutil.
Note: this change also makes autoninja always make a decision about
-j; there's no longer a default where it lets ninja pick. The code is
simpler this way and I think it's better because it lets developers
always see which -j is in effect when using autoninja (and that's its
exact purpose, if you wanted default you shouldn't have used autoninja).
R=dpranke@chromium.org
Bug: 976265
Change-Id: Ic9d12469729e4bf58da1ec1bd70437329519fc46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1663904
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Gabriel Charette <gab@chromium.org>
Auto-Submit: Gabriel Charette <gab@chromium.org>
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
Regex strings need to be raw or else double-escape all the \ characters,
but Python 2.7 doesn't reliably enforce this, so bugs have crept in.
The buggy regex from create_installer_archive.py that this was copied
from was deleted a while ago. However this bad pattern is pervasive in
Chrome and depot_tools, as crudely found by searching for:
re\.[a-z]*\('.*\\[dsw\(\.]
An earlier instance of this was found when running some of our scripts
with Python 3.8.
Bug: 958138
Change-Id: If7ded5ae13f8cc36a5f6277c6ae0a2f54f88c3e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1590191
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Large parallelism causes too many open files error.
Let me limit to 500 for Mac now.
Bug: 936864
Change-Id: I2fccc8cf14483c6f34d84c84d82c44df6e4f3177
Reviewed-on: https://chromium-review.googlesource.com/c/1496675
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
On Windows, without quote, whole return value would be treated as
a command, and the execution would fail.
On Linux and Mac, without quote, if depot_tools is settled under
directories with ' ', command execution would fail because paths are
separated in a wrong way.
To make such a return value work on Linux and Mac, the shell script
started to use eval.
Bug: 902930
Change-Id: I9bb74585294af565988c0b844b6b113a5c685530
Reviewed-on: https://chromium-review.googlesource.com/c/1325249
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Since ninja uses getopt_long to parse the argument, we need to suppose
-j100 and -tclean.
Please see ReadFlags function in:
https://github.com/ninja-build/ninja/blob/master/src/ninja.cc
Change-Id: If61f05e66be546a591549f6f153cce994c972309
Reviewed-on: https://chromium-review.googlesource.com/c/1335428
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When people execute autoninja, I believe people would expect ninja
in the autoninja directory would be used. However, the original code
find ninja from PATH, and ninja in other directory could be used.
People usually do not notice this because having depot_tools in
PATH is recommended for Chromium developers. However, in some
environments, an old version ninja is pre-installed, and unexpected
version ninja could be used upon PATH environment.
Also, in most of bots, depot_tools directory is not included in PATH.
Autoninja execution would fail there because the system cannot find
ninja.
Bug: b/77176746
Change-Id: Iad8bd952dc1e34a9d303fd5b493c555156369a17
Reviewed-on: https://chromium-review.googlesource.com/c/1319489
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Auto-Submit: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
autoninja automatically sets -l <num_cores>. -l option makes ninja
not invoke a new command if current CPU load average is above -l.
However, as far as I investigated, -l <num_cores> make the build
much slower than without -l especially on Linux & OSX machines with
small numbers of cores. I should say -l decreases the build
performance. When I build Chromium with Goma with the same -j with
autoninja without -l, the load average goes more than the number of
cores while keeping the machine working as usual.
Also, ninja can invoke commands until the spike of command invocation
is reflected to load average, -l might not mitigate for a machine
to get stuck by too high load.
Note that from what I understand from the implementation, Windows
ninja's posix-compatible load average is always less than num_cores.
i.e. -l <num_cores> won't limit the process invocation.
5984986459/src/util.cc (L479)
Let me make autoninja not set -l and bring better build performance
on machines with small number of cores.
Bug: b/117810340
Change-Id: I50f231f1a8976f8ecfc3a0c778f0f1ac98d3827f
Reviewed-on: https://chromium-review.googlesource.com/c/1290611
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Nina supports -C out/Default and -Cout/Default to specify the build
directory so autoninja should also. This change adds that support.
Bug: 890744
Change-Id: I5e824242ed4b333ac99f1ee9a649ffcfa03a812e
Reviewed-on: https://chromium-review.googlesource.com/c/1257586
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When GOMA_DISABLED is set goma will use the local compiler. Autoninja
needs to understand this in order to avoid requesting too much
parallelism.
Change-Id: Ic124893dd583a401d0d9ad7fbd27ee9b6715fcfe
Reviewed-on: https://chromium-review.googlesource.com/1015402
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
This is the second attempt at fixing autoninja to allow passing '^^' to
it to specify that ninja should build the outputs of the specified file
instead of building that file itself. The problem is that '^' is a
special character and when extra layers of indirection are added the
number of '^' characters needed grows exponentially in some poorly
understood way. The first fix attempt just quoted the arguments that
autoninja.bat passed to autoninja.py, but that meant they came in as
one argument. This fix expands on that by modifying autoninja.py to
understand how to deal with the monolithic argument. With this change
this once again works:
autoninja -C out\debug_component ..\..\base\win\enum_variant.cc^^
It can be convenient to have a ninja.bat file which starts goma and lets
users keep typing the same build commands. However even with this fix
the previously recommended ninja.bat file must be invoked with four
'^' characters. If that is too much then the new recommended ninja.bat
is to copy autoninja.bat and modify as needed, perhaps like this:
@echo off
call python c:\goma\goma-win64\goma_ctl.py ensure_start >nul
FOR /f "usebackq tokens=*" %%a in (`python c:\src\depot_tools\autoninja.py "%*"`) do echo %%a & %%a
BUG: 758725
Change-Id: Ieee9cf343ee5f22e9988a1969cb7a7a90687666b
Reviewed-on: https://chromium-review.googlesource.com/656478
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
The -t tools in ninja fail if -j or -l are specified. So, autoninja.py
needs to watch for -t and omit -j and -l if it is noticed.
R=sebmarchand@chromium.org
BUG=740227
Change-Id: I1418193daeab154178d15be60ab09551bacaf3af
Reviewed-on: https://chromium-review.googlesource.com/563775
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Using goma requires the developer to remember which build directories
use goma and which don't so that they can pass an appropriate -j number.
Getting this wrong makes builds slower, either by under utilizing
resources or by causing a self-inflicted DOS attack. Usage:
autoninja -C out/debug
autoninja looks at the settings for the specified build directory and
then selects either -j num_cores*20 or no -j flag based on the
use_goma setting.
You can set the NINJA_CORE_MULTIPLIER variable to change from the
default 20* multiplier. You can also use NINJA_CORE_ADDITION if you
want non-goma builds to specify -j with an offset to the number of
cores, such as this Linux command:
NINJA_CORE_ADDITION=-2 autoninja -C out/release base
This will tell autoninja to pass -j to ninja with num_cores-2 as the
parameter.
On Windows you can have a ninja.bat file (ahead of ninja on the path)
such that autoninja will automatically be used. It should contain this:
@call autoninja.bat %*
Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655
Reviewed-on: https://chromium-review.googlesource.com/517662
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Sam Clegg
Jesse McKenna
Gabriel Charette
Raul Tambre
Bruce Dawson
Takuto Ikuta
Yoshisato Yanagisawa
Allen Bauer