depot_tools/autoninja.bat

@echo off
:: Copyright 2017 The Chromium Authors. All rights reserved.
:: Use of this source code is governed by a BSD-style license that can be
:: found in the LICENSE file.

setlocal

REM Set unique build ID.
FOR /f "usebackq tokens=*" %%a in (`python -c "from __future__ import print_function; import uuid; print(uuid.uuid4())"`) do set AUTONINJA_BUILD_ID=%%a

REM If a build performance summary has been requested then also set NINJA_STATUS
REM to trigger more verbose status updates. In particular this makes it possible
REM to see how quickly process creation is happening - often a critical clue on
REM Windows. The trailing space is intentional.
if "%NINJA_SUMMARIZE_BUILD%" == "1" set NINJA_STATUS=[%%r processes, %%f/%%t @ %%o/s : %%es ] 

REM Execute whatever is printed by autoninja.py.
REM Also print it to reassure that the right settings are being used.
FOR /f "usebackq tokens=*" %%a in (`python %~dp0autoninja.py "%*"`) do echo %%a & %%a
@if errorlevel 1 goto buildfailure

REM Use call to invoke python script here, because we use python via python.bat.
@if "%NINJA_SUMMARIZE_BUILD%" == "1" call python.bat %~dp0post_build_ninja_summary.py %*
@call python.bat %~dp0ninjalog_uploader_wrapper.py --cmdline %*

exit /b
:buildfailure

@call python.bat %~dp0ninjalog_uploader_wrapper.py --cmdline %*

REM Return an error code of 1 so that if a developer types:
REM "autoninja chrome && chrome" then chrome won't run if the build fails.
cmd /c exit 1
autoninja - ninja wrapper to make goma easier Using goma requires the developer to remember which build directories use goma and which don't so that they can pass an appropriate -j number. Getting this wrong makes builds slower, either by under utilizing resources or by causing a self-inflicted DOS attack. Usage: autoninja -C out/debug autoninja looks at the settings for the specified build directory and then selects either -j num_cores20 or no -j flag based on the use_goma setting. You can set the NINJA_CORE_MULTIPLIER variable to change from the default 20 multiplier. You can also use NINJA_CORE_ADDITION if you want non-goma builds to specify -j with an offset to the number of cores, such as this Linux command: NINJA_CORE_ADDITION=-2 autoninja -C out/release base This will tell autoninja to pass -j to ninja with num_cores-2 as the parameter. On Windows you can have a ninja.bat file (ahead of ninja on the path) such that autoninja will automatically be used. It should contain this: @call autoninja.bat %* Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655 Reviewed-on: https://chromium-review.googlesource.com/517662 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 8 years ago			`@echo off`
			`:: Copyright 2017 The Chromium Authors. All rights reserved.`
			`:: Use of this source code is governed by a BSD-style license that can be`
			`:: found in the LICENSE file.`

Set NINJA_STATUS for better build performance summarizing When NINJA_SUMMARIZE_BUILD=1 then autoninja prints a summary of build performance at the end. This change makes it so that ninja prints more detailed build statistics during the build. In particular, slow process creation is a common but difficult to see bottleneck. This change makes it so that if NINJA_SUMMARIZE_BUILD=1 then the number of running build processes will be displayed. Bug: 787983 Change-Id: Ic0907e23e7f762e23e4795059b37085e0cb8c4ea Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1582802 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 6 years ago			`setlocal`

Set unique build ID. For ease of distinguishing a series of build from others in Goma, let me make autoninja set unique build ID. The ID can be usable for statistics purpose to see how much time is usually spent for each build. Since the unique ID is not specific to Goma, i.e. anybody invoked by ninja can use it to distinguish builds, I chose AUTONINJA_BUILD_ID. Since we do not need to distinguish each node, I chose uuid4. Bug: b/77176746 Change-Id: I0b6a67811b420c51d1fd3fd113bf7f039a41e2ab Reviewed-on: https://chromium-review.googlesource.com/c/1275685 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org> 6 years ago			`REM Set unique build ID.`
Convert print statements to Python 3 style Ran "2to3 -w -n -f print ./" and manually added imports. Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually. Also manually added imports to files, which used print as a function, but were missing the import. The scripts still work with Python 2. There are no intended behaviour changes. Bug: 942522 Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Dirk Pranke <dpranke@chromium.org> Auto-Submit: Raul Tambre <raul@tambre.ee> 6 years ago			FOR /f "usebackq tokens=*" %%a in (`python -c "from __future__ import print_function; import uuid; print(uuid.uuid4())"`) do set AUTONINJA_BUILD_ID=%%a
Set unique build ID. For ease of distinguishing a series of build from others in Goma, let me make autoninja set unique build ID. The ID can be usable for statistics purpose to see how much time is usually spent for each build. Since the unique ID is not specific to Goma, i.e. anybody invoked by ninja can use it to distinguish builds, I chose AUTONINJA_BUILD_ID. Since we do not need to distinguish each node, I chose uuid4. Bug: b/77176746 Change-Id: I0b6a67811b420c51d1fd3fd113bf7f039a41e2ab Reviewed-on: https://chromium-review.googlesource.com/c/1275685 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Yoshisato Yanagisawa <yyanagisawa@chromium.org> 6 years ago
Set NINJA_STATUS for better build performance summarizing When NINJA_SUMMARIZE_BUILD=1 then autoninja prints a summary of build performance at the end. This change makes it so that ninja prints more detailed build statistics during the build. In particular, slow process creation is a common but difficult to see bottleneck. This change makes it so that if NINJA_SUMMARIZE_BUILD=1 then the number of running build processes will be displayed. Bug: 787983 Change-Id: Ic0907e23e7f762e23e4795059b37085e0cb8c4ea Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1582802 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 6 years ago			`REM If a build performance summary has been requested then also set NINJA_STATUS`
			`REM to trigger more verbose status updates. In particular this makes it possible`
			`REM to see how quickly process creation is happening - often a critical clue on`
			`REM Windows. The trailing space is intentional.`
			`if "%NINJA_SUMMARIZE_BUILD%" == "1" set NINJA_STATUS=[%%r processes, %%f/%%t @ %%o/s : %%es ]`

autoninja - ninja wrapper to make goma easier Using goma requires the developer to remember which build directories use goma and which don't so that they can pass an appropriate -j number. Getting this wrong makes builds slower, either by under utilizing resources or by causing a self-inflicted DOS attack. Usage: autoninja -C out/debug autoninja looks at the settings for the specified build directory and then selects either -j num_cores20 or no -j flag based on the use_goma setting. You can set the NINJA_CORE_MULTIPLIER variable to change from the default 20 multiplier. You can also use NINJA_CORE_ADDITION if you want non-goma builds to specify -j with an offset to the number of cores, such as this Linux command: NINJA_CORE_ADDITION=-2 autoninja -C out/release base This will tell autoninja to pass -j to ninja with num_cores-2 as the parameter. On Windows you can have a ninja.bat file (ahead of ninja on the path) such that autoninja will automatically be used. It should contain this: @call autoninja.bat %* Change-Id: I4003e3fc323d1cbab612999c945b5a8dc5bc6655 Reviewed-on: https://chromium-review.googlesource.com/517662 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 8 years ago			`REM Execute whatever is printed by autoninja.py.`
			`REM Also print it to reassure that the right settings are being used.`
Fix autoninja to allow compiling one source file This is the second attempt at fixing autoninja to allow passing '^^' to it to specify that ninja should build the outputs of the specified file instead of building that file itself. The problem is that '^' is a special character and when extra layers of indirection are added the number of '^' characters needed grows exponentially in some poorly understood way. The first fix attempt just quoted the arguments that autoninja.bat passed to autoninja.py, but that meant they came in as one argument. This fix expands on that by modifying autoninja.py to understand how to deal with the monolithic argument. With this change this once again works: autoninja -C out\debug_component ..\..\base\win\enum_variant.cc^^ It can be convenient to have a ninja.bat file which starts goma and lets users keep typing the same build commands. However even with this fix the previously recommended ninja.bat file must be invoked with four '^' characters. If that is too much then the new recommended ninja.bat is to copy autoninja.bat and modify as needed, perhaps like this: @echo off call python c:\goma\goma-win64\goma_ctl.py ensure_start >nul FOR /f "usebackq tokens=" %%a in (`python c:\src\depot_tools\autoninja.py "%"`) do echo %%a & %%a BUG: 758725 Change-Id: Ieee9cf343ee5f22e9988a1969cb7a7a90687666b Reviewed-on: https://chromium-review.googlesource.com/656478 Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 8 years ago			FOR /f "usebackq tokens=" %%a in (`python %~dp0autoninja.py "%"`) do echo %%a & %%a
Script to summarize performance of ninja builds post_build_ninja_summary.py analyzes the .ninja_log file in an output directory to summarize build performance. It lists the most expensive build steps, and build-step types, printing both their elapsed time and, more importantly, their "weighted" time which is a measure of their contribution to the total build time. For each time segment within the build the running build steps accumulate weighted time that is proportional to the elapsed time divided by the number of tasks running. If a thousand compilation steps are running in parallel then they will each be "charged" for 1/1,000th of the elapsed time. If a single link step is running alone then it is charged with all of the elapsed time. Bug: chromium:787983 Change-Id: Id5aea715f798a16415dd0365a27f0051202668e5 Reviewed-on: https://chromium-review.googlesource.com/871988 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Erik Staab <estaab@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 7 years ago			`@if errorlevel 1 goto buildfailure`
introduce ninjalog_uploader to autoninja.bat This is windows version of https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1345255 Changed to use call to invoke python because python is bat file. Bug: 900161 Change-Id: I0332306387614153be7ae6dbde05fb2bc93559b1 Reviewed-on: https://chromium-review.googlesource.com/c/1355641 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> 6 years ago
			`REM Use call to invoke python script here, because we use python via python.bat.`
			`@if "%NINJA_SUMMARIZE_BUILD%" == "1" call python.bat %~dp0post_build_ninja_summary.py %*`
[ninjalog] re-enable log upload I got launch approval. http://shortn/_dyJK6ME7C0 Detailed document of this log upload is in https://chromium.googlesource.com/chromium/tools/depot_tools/+/HEAD/ninjalog.README.md Bug: 900161 Change-Id: Iad2f954040d40caa5b7cc1f1eb39c1088f7f316e Reviewed-on: https://chromium-review.googlesource.com/c/1401919 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@chromium.org> 6 years ago			`@call python.bat %~dp0ninjalog_uploader_wrapper.py --cmdline %*`
introduce ninjalog_uploader to autoninja.bat This is windows version of https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1345255 Changed to use call to invoke python because python is bat file. Bug: 900161 Change-Id: I0332306387614153be7ae6dbde05fb2bc93559b1 Reviewed-on: https://chromium-review.googlesource.com/c/1355641 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> 6 years ago
Fix autoninja && chrome on failed builds A handy usage pattern for autoninja.bat that I was not aware of is to go autoninja -C out\Default chrome && chrome This will build chrome and then run it, but only run it if the build succeeds. The addition of post_build_ninja_summary.py broke this by losing the error code. This change fixes it by using black magic to set an error code in the failure case. Bug: chromium:787983 Change-Id: Ib87fd1799816e19d56de76e08e0f9688be903d80 Reviewed-on: https://chromium-review.googlesource.com/916705 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 7 years ago			`exit /b`
Script to summarize performance of ninja builds post_build_ninja_summary.py analyzes the .ninja_log file in an output directory to summarize build performance. It lists the most expensive build steps, and build-step types, printing both their elapsed time and, more importantly, their "weighted" time which is a measure of their contribution to the total build time. For each time segment within the build the running build steps accumulate weighted time that is proportional to the elapsed time divided by the number of tasks running. If a thousand compilation steps are running in parallel then they will each be "charged" for 1/1,000th of the elapsed time. If a single link step is running alone then it is charged with all of the elapsed time. Bug: chromium:787983 Change-Id: Id5aea715f798a16415dd0365a27f0051202668e5 Reviewed-on: https://chromium-review.googlesource.com/871988 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Erik Staab <estaab@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 7 years ago			`:buildfailure`
[autoninja] upload ninjalog even in failed build I noticed current autoninja does not upload build log in failed build. Change-Id: Ie58646b483e130769ad22113953c0b95b3145548 Reviewed-on: https://chromium-review.googlesource.com/c/1441892 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@chromium.org> 6 years ago
			`@call python.bat %~dp0ninjalog_uploader_wrapper.py --cmdline %*`

Fix autoninja && chrome on failed builds A handy usage pattern for autoninja.bat that I was not aware of is to go autoninja -C out\Default chrome && chrome This will build chrome and then run it, but only run it if the build succeeds. The addition of post_build_ninja_summary.py broke this by losing the error code. This change fixes it by using black magic to set an error code in the failure case. Bug: chromium:787983 Change-Id: Ib87fd1799816e19d56de76e08e0f9688be903d80 Reviewed-on: https://chromium-review.googlesource.com/916705 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> 7 years ago			`REM Return an error code of 1 so that if a developer types:`
			`REM "autoninja chrome && chrome" then chrome won't run if the build fails.`
			`cmd /c exit 1`