Commit Graph

55 Commits (d1259b49c5b41bc4939ba7c5bb8e8e08083a05f6)

Author SHA1 Message Date
Junji Watanabe d3d4c0ac53 autoninja: Use AUTONINJA_BUILD_ID for RBE_invocation_id as is.
To use build id/invocation id consistently, reclient_helper will set AUTONINJA_BUILD_ID as RBE_invocation_id as is.
The host name information will be added as a metrics label.

Bug: 355127782
Change-Id: I5609ef38d897b19029795adf1b288898986c7c70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5737676
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
9 months ago
Junji Watanabe b12257963c autoninja: Replace ninjalog_uploader_wrapper.py and reclient_metrics.py with build_telemetry.py
This CL unifies the opt-in/opt-out handling for build telemetry collections about Reclient and Ninjalog.

The user consent message will be displayed only once at the beginning of a build.

```
❯ autoninja -C out/deterministic-andorid-dbg base
*** NOTICE ***
Google-internal telemetry (including build logs, username, and hostname) is collected on corp machines to diagnose performance and fix build issues. This reminder will be shown 9 more times. See http://go/chrome-build-telemetry for details. Hide this notice or opt out by running: build_telemetry [opt-in] [opt-out]
*** END NOTICE ***

Proxy started successfully.
...
```

Bug: 345113094
Change-Id: Ie5886287c4bd20262be0ff247508ac3869441eb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5669094
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
9 months ago
Takuto Ikuta 791894e4c4 reclient_helper: tuning local execution parameter
With this CL, build time of chrome with non-cache hit case improved a
bit on P620 Linux workstation.

```
Benchmark 1: RBE_local_resource_fraction=0.4 RBE_racing_bias=0.7 RBE_remote_accept_cache=false autoninja -C out/Default chrome
  Time (mean ± σ):     337.697 s ± 11.439 s    [User: 3086.301 s, System: 1362.952 s]
  Range (min … max):   327.732 s … 351.680 s    5 runs
```

```
Benchmark 1: RBE_remote_accept_cache=false autoninja -C out/Default chrome
  Time (mean ± σ):     347.795 s ± 15.906 s    [User: 2764.181 s, System: 1199.244 s]
  Range (min … max):   331.964 s … 367.898 s    5 runs
```

Bug: 352463976
Change-Id: Ic9732583857ce175888a7ec65cf9e323529f0a07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5693250
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
10 months ago
Junji Watanabe 1387a8c958 autoninja: Refactor the code calling Reclient
- Remove reclient_helper.py to avoid calling it directly.
- Cleans up autoninja.py

I just want to do this cleanup before merging the user notice messages
for logs/metrics collection.

Bug: 345113094
Change-Id: I0c76426b3cb387eae6dede031727c107e57d5d1a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5668282
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
10 months ago
Louis Romero 4d9ad255f1 Remove a space between a value and its unit
Prior to this change, autoninja output was:

```
src/buildtools/reclient/bootstrap --re_proxy=src/buildtools/reclient/reproxy --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
Proxy started successfully.
0.705 s to start reproxy
 2.85s loading fs state
 2.80s load build.ninja
ninja: no work to do.
7.38s Build Succeeded: 0 steps - 0.00/s
src/buildtools/reclient/bootstrap --shutdown --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
RBE Stats: down 0 B, up 0 B,
0.331 s to stop reproxy
```

Now it's:

```
src/buildtools/reclient/bootstrap --re_proxy=src/buildtools/reclient/reproxy --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
Proxy started successfully.
0.705s to start reproxy
 2.85s loading fs state
 2.80s load build.ninja
ninja: no work to do.
7.38s Build Succeeded: 0 steps - 0.00/s
src/buildtools/reclient/bootstrap --shutdown --cfg=src/buildtools/reclient_cfgs/reproxy.cfg
RBE Stats: down 0 B, up 0 B,
0.331s to stop reproxy
```

Change-Id: If1c4cd211f29267ed01c14efbe2fe09a1d101287
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5509391
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Louis Romero <lpromero@google.com>
12 months ago
Michael Savigny f8d1ff28d2 Override reclient timeouts on windows.
Set the timeouts in autoninja for now, determine template rollout
problem after.

Bug: b/335525655
Change-Id: I0021b3031a9e8952acea465af94f61a4c50d774f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5477047
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 917a57459b Turn off creds caching for windows devs.
Windows devs use luci-auth as the auth mechanism (as credshelper).  By
default, credshelper-based auth is cached, but that doesn't work quite
correctly with luci-auth (it does it's own caching).  It can cause
intermittent auth failures during a build, which can slow down build
performance.

Bug: b/335525655
Change-Id: Ic4ef25cfa8e0f37929170e306e8bf57e737a4a72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5473742
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
1 year ago
Michael Savigny 0d48b75661 Set windows devs to use remote-local-fallback.
There are performance issues with racing finalization that need to be
investigated.

Bug:b/335525655
Change-Id: Ic7e7e3d34f3ce6e75d4939ff3ba2fc3cf734a99d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5464353
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
1 year ago
Michael Savigny 65faa11896 Revert "[reclient] Limit local resource tuning on windows to cloudtops"
This reverts commit 90d54ff344.

Reason for revert: b/335525655 Racing with higher resource fraction on physical windows machines seems to also be a problem.

Original change's description:
> [reclient] Limit local resource tuning on windows to cloudtops
>
> Bug: b/330173550
> Change-Id: I65ce93eeb216a7a4b77506fbcb515697fdecde2c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420670
> Commit-Queue: Ben Segall <bentekkie@google.com>
> Reviewed-by: Michael Savigny <msavigny@google.com>

Bug: b/330173550
Change-Id: I94479584bd2a041e0e621a8cf3f3cb7f65600a57
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5462390
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
1 year ago
Ben Segall 90d54ff344 [reclient] Limit local resource tuning on windows to cloudtops
Bug: b/330173550
Change-Id: I65ce93eeb216a7a4b77506fbcb515697fdecde2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420670
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 083ed2e307 Ignore errors on racing cleanup.
Occasionally on windows there seems to be a racing problem where rmtree
tries to delete something that no longer exists.  This seems to happen
when a build is cancelled/terminated while in-progress, so racing
actions in reproxy might have deleted the temporary file while autoninja
is cleaning up stuff.  In any case, missing a file in this case isn't
something to fail over.

Bug: b/329738642
Change-Id: I86240e31093c6415707878277b01db5931b175ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376532
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Philipp Wollermann 43bedd368b reclient_helper.py: Simplify workspace_is_on_slow_filesystem
I initially didn't want to make this code Google-specific, however I now
believe that it's better for everyone to simplify it like this:

- It's much less code to just check for the path prefix. We do this
  already in gclient.py, too, so the approach is proven to work.
- The call to findmnt introduces a 15ms overhead that affects everyone,
  even users who will never have their workspace on Cog.
- External folks who run their own infrastructure likely have their own
  fork of depot_tools anyway and can patch this as needed.

Bug: b/324547324
Change-Id: Ic00d417c280ac71529d106cfb2bcc844dfd44bd3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5358773
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
1 year ago
Ben Segall 1a4a96ccdc Add link to documentation if reproxy fails to start
Bug: b/328269200
Change-Id: I647d8ae222ab073324ce7bf1ad4680242526882a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5345028
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
1 year ago
Philipp Wollermann 19187e7927 reclient_helper.py: Fix error on macOS < 14.0
The `-Y` flag was added to `df` only in the latest macOS version, so we
should guard its usage with a version check.

Change-Id: Ifafe47de5f9e86ce6a4605a1adb26994db5f33d9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5345461
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
1 year ago
Philipp Wollermann 771b8d7983 reclient_helper.py: Disable racing on slow filesystems
Bug: b/324547324
Change-Id: I64c22d69ec947345cc594ea3e122c33f58d21053
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5339976
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Aleksandr Mikhailov <avmikhailov@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
1 year ago
Michael Savigny 9d7c8e76f8 Change source of hash for reproxy socket.
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory.  Should allow concurrent builds against the same output
directory as much as ninja allows it.

Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
1 year ago
Michael Savigny f76550541c Redo of racing temp directory.
Change is that racing directory is deleted atexit instead of being
cleaned up in a sweep at the start, whick wouldn't work anyway for the
actual use case the change was for (in-use racing directories would get
deleted, breaking concurrent builds).

Bug:b/324877315
Change-Id: I3baf25b07c13061e9b4c14a3a701ed9eceb9bbb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5288415
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
1 year ago
Michael Savigny 2acd6f4c9d Revert "Changing location of temp racing directory."
This reverts commit 566c3772a0.

Reason for revert: Seems to be causing flaky failures in win-trace angle try builder, see b/323830218 for details.

Original change's description:
> Changing location of temp racing directory.
>
> Put temporary racing artifacts into
> .reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
> simultaneous builds against the same output directory.
>
> Bug: b/321554715
> Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
> Reviewed-by: Junji Watanabe <jwata@google.com>
> Reviewed-by: Ben Segall <bentekkie@google.com>

Bug: b/321554715
Change-Id: Ib6fa285bb435d7c0d82960633364669e6575d73b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286354
Commit-Queue: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 566c3772a0 Changing location of temp racing directory.
Put temporary racing artifacts into
.reproxy_tmp/racing/<timestamp>-<hash> in prep for permitting
simultaneous builds against the same output directory.

Bug: b/321554715
Change-Id: Iae9af9aed03197184e6d11da010d0bb10779ed23
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5273209
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
1 year ago
Michael Savigny 7142bf166c Enable deps cache for mac devs.
Turns on deps cache, and increases it's size, for mac developers.  In
some cases, it shows good improvements on chrome builds (cutting 1-4
minutes off a full chrome build, once populated).

Bug: b/322801289
Change-Id: I9cd2b35b5429228c07b0703d6c3344963e4fb9e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5258563
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny b482a5dbba Reduce local resource fraciton for windows builds.
Reduces resources used for local compilation in racing for windows.
Makes machine running the build more responsive during the build.
Testing seems to show it doesn't impact overall build performance, and might even improve it slightly.

Bug: b/309819402
Change-Id: I6f4f41887b165c753508a57fcaa1a1a6931feee3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254559
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Auto-Submit: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny 10bd39fd47 Enable deps cache for windows builds.
On my windows cloudtop, without deps cache a build of chrome took around 2130s, with it enabled (and populated) a build took around 1665s, which is about a 7-8 minute improvement.


Bug: b/320457907
Change-Id: I82232cad57152f1a10d1e262125290d54bf8ab33
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5249053
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Michael Savigny 8d97b6d89b Add mac-specific reclient tweak.
Set cas_concurrency for mac builds to 100.  It doesn't affect
performance on high-speed networks, but has shown an improvement on
networks prone to congestion.

Note that these adjustments have been checked out for builds on macs, and the cas_concurrency value being a default of 100 for mac doesn't mean it should be the default for other oses.

Bug: b/304616237
Change-Id: I9431fdb35737f9f2b236296625af2f2a31e52754
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5075857
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Michael Savigny af69249965 Fix deletion of old build logs
The code rotating the old build logs expects to delete directories
(since it uses rmtree) but tries to delete all items returned by
os.listdir(...) which may includes symbolic links or regular files.

Change it to only consider the old build logs directories instead.

Based on crrev/5028323, with fix to unit test.

Fixed: b/310900283
Change-Id: I4c618e6618c0193331c063028ebf02d8c4e7baee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5038467
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
1 year ago
Anas Sulaiman 1e503bf950 don't confuse users with unimportant and non-actionable error msg
Bug: b/310852966
Change-Id: I2c64a92a36b817043ff1f7abfa528d4c7aa449e8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5030272
Reviewed-by: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Anas Sulaiman <mrahs@google.com>
1 year ago
Anas Sulaiman 7a9b709a6c avoid unnecessary failures in get_hostname
The catch clause was specific to one error type, but other errors may also happen such as socket.herror. In any case, the point is to fallback anyways.

Bug: b/310216577
Change-Id: I55561f8f40446f90bee27ea2aab1a4e60c03e7e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5024019
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Anas Sulaiman <mrahs@google.com>
Reviewed-by: Michael Savigny <msavigny@google.com>
1 year ago
Takuto Ikuta 5388959fd7 reclient_helper: delete directory synchronously
This is for
https://crrev.com/c/4916355/8/reclient_helper.py#177

Let me use synchronous directory deletion as reclient doesn't produce
many log files.

Change-Id: I44f616c9d825bc6b9dbeea329ca1db00a578a9fb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4988347
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Michael Savigny <msavigny@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
2 years ago
Ben Segall 27ea34f94e [reclient] Keep previous 5 builds worth of logs
Test: Updated unit tests

Bug: b/300945159
Change-Id: Id0ce3471be17ab05dbef8e3091333149bd9bde6a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4916355
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
2 years ago
Ben Segall 1077fbe08a [reclient] Only show shutdown msg if ctrl-C end the build
Bug: b/300284667
Change-Id: I28151a974cd20cb5ec7a789cf7f7cd0c926b4486
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864734
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall c18c91eca1 [reclient] Only print reproxy start/stop times if NINJA_SUMMARIZE_BUILD=1
Bug: b/300284667
Change-Id: Ic8a41b573aa3ec4b2e852fbfe7455b941cf9fec3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4864442
Commit-Queue: Ramy Medhat <abdelaal@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
2 years ago
Ben Segall b64ee7f525 [reclient] Add hostname to invocation ids
This is already covered in the PDD for googlers. We are already gathering hostnames in cloud monitoring mertrics


Bug: b/294945709
Change-Id: Idfb5b8d3b7cf64edd60092f6bd49a3f9f1950d15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4848094
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 29282b581b [reclient] Add cred cache status as a metrics label
This will help analyze startup metrics

Bug: b/294945709
Change-Id: Ia397dbebef7cc30b49c614dda51e56f482b6145b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4847325
Commit-Queue: Ben Segall <bentekkie@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Junji Watanabe 2d109056fb [reclient] Add quotes to download_remoteexec_cfg in error message
Fixed: b/295319793
Change-Id: Ida0c7a01a04d396bc36f64742f973a75b11fb0f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4770874
Auto-Submit: Junji Watanabe <jwata@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Junji Watanabe 0f8ce6d99f [reclient] update error message for missing reclient binaries/configs
`Developer builds with reclient are not yet supported` is misleading.
This CL updates the message to help users to resolve the issues.

Bug: b/294933473
Change-Id: Ibce2854e25b8994af58011f4e7661f467fe910e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4755000
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Auto-Submit: Junji Watanabe <jwata@google.com>
2 years ago
Bruce Dawson 6d0c235dae Print reproxy start/stop times
On NOP or incremental builds the reproxy startup/shutdown times can be a
significant portion of the total build time and it is important to
acknowledge these times.

Bug: b/294443556
Change-Id: Ic63331683632c8f447858b498f53d254b56ffe89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4753472
Reviewed-by: Philipp Wollermann <philwo@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Ben Segall 582edfab2c [reclient] Dont limit local resources if RBE_remote_disabled=1
Bug: b/288564971
Change-Id: Ia3e556e5b30ca4ca82219e5ee696b7c982419de3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4748579
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Commit-Queue: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall d3e43dd431 [reclient] Set some sensible defaults for racing that match goma
local_resource_fraction is set to 0.2 as goma limited the max local
racing subprocesses to num_cpu/5

racing_bias is set to 0.95 as experimentally that ensures reasonable
behaviour for cache hits.

This cl does not enable racing by default. These values are only used if
RBE_exec_strategy=racing is set

Test: Tested builds on my mac and linux cloudtop and got mostly remote exec
and cache hits with some local executions
Bug: b/288564971
Change-Id: Idb71c03f4addd23d0253dc5bf3564058cd212284
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4713186
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
2 years ago
Ben Segall 7899866866 [reclient] Dont fail build if previous reproxy instance failed to shutdown
This is caused by the previous reproxy instance still having the log files open for writing. Bootstrap will kill that instance on startup of the next build but that only happens after we try and create the log dir

Bug: b/290230382
Change-Id: Idd88cfa9c571c0bca078f8bf0078fdb3941603ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4670328
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 3589c48471 [reclient] Workaround to remove mdproxy prodcertstatus from path
mdproxy/bin/prodcertstatus confuses the automatic auth logic in reproxy
This will be fixed in the next release but for now we remove mdproxy
from the path for autoninja builds

Test: Installed mdproxy locally and verified that gcert is not called
Change-Id: I0912a3c8b00a0cda41a85a16f51b8f54b20f1974
Bug: b/292523514
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4712465
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall d411904b84 [reclient] disable racing for now until it is less agressive
Bug: b/288564971
Change-Id: I06d1421ff7de353d6e87be33ac3dca8e06ff6880
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4704566
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Andus Yu <andusyu@google.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 60b21dd193 [reclient] Add 'exist_ok=True' for racing_tmp_dir to handle subsequent builds
Bug: b/288285261
Change-Id: Ied87152e6af0782739bdaa0d37a56ec783f66e33
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4698963
Commit-Queue: Ben Segall <bentekkie@google.com>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 04ca383754 Reland "reclient: Enable racing for chrome developers"
This reverts commit 3aa7489c3e.

Reason for revert: reproxy 0.110.0 includes a flag for fixing hardlink issue

Original change's description:
> Revert "reclient: Enable racing for chrome developers"
>
> This reverts commit 24ac2543c7.
>
> Reason for revert: hard links dont work on tmpfs b/288997481
>
> Original change's description:
> > reclient: Enable racing for chrome developers
> >
> > racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.
> >
> > Bug: b/288285261
> > Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
> > Commit-Queue: Ben Segall <bentekkie@google.com>
> > Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
>
> Bug: b/288285261
> Change-Id: I7b262f63d4edd68447fcec0a781ef55923635d2c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4651968
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Ben Segall <bentekkie@google.com>

Bug: b/288285261
Change-Id: I8fb65c32adc7b22a747c30f408f1f34b2bb5f8f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4693625
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 3aa7489c3e Revert "reclient: Enable racing for chrome developers"
This reverts commit 24ac2543c7.

Reason for revert: hard links dont work on tmpfs b/288997481

Original change's description:
> reclient: Enable racing for chrome developers
>
> racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.
>
> Bug: b/288285261
> Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
> Commit-Queue: Ben Segall <bentekkie@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>

Bug: b/288285261
Change-Id: I7b262f63d4edd68447fcec0a781ef55923635d2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4651968
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ben Segall <bentekkie@google.com>
2 years ago
Ben Segall 24ac2543c7 reclient: Enable racing for chrome developers
racing_bias=0.95 is a best guess based on benchmarks on my glinux and gwindows cloudtops, and my gmac laptop. I will monitor build performance data and adjust the bias based on that.

Bug: b/288285261
Change-Id: Ie8652281f7bc815e64978bcd56a75e5ec28e2132
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4633180
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
2 years ago
Ben Segall 2e6738447e reclient: Clear log dir between builds
Bug: b/287650736
Change-Id: I87c0226f428211ac99520b024edeaf5ccb6a674c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4629861
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Ben Segall <bentekkie@google.com>
2 years ago
Fumitoshi Ukai dedeb88991 reclient: show warning if $RBE_instance is set
Bug: b/287174095
Change-Id: Ia8c89e36490e4a204582c566ad654fcba9aa7aaa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4616940
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Richard Wang <richardwa@google.com>
Auto-Submit: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Richard Wang <richardwa@google.com>
2 years ago
Ben Segall e49349becf Mark reclient build metrics with the tool that started reclient
This will allow us to segment siso and ninja metrics for analysis

Bug: b/281504726
Change-Id: If45b272004b236d634c717ea9e28e38e49c6231a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4568610
Commit-Queue: Ben Segall <bentekkie@google.com>
Auto-Submit: Ben Segall <bentekkie@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
2 years ago
Ben Segall 530d86d40b Implement reclient metrics uploading
This cl will be submitted after the cl to add the metadata to the
reproxy.cfg file has been submitted to chromium/src: crrev/c/4513215

Bug: b/281504726
Change-Id: Ifa6d5f56d4a85ccb9ec8e4f70207d8b6b9382e89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4516023
Commit-Queue: Ben Segall <bentekkie@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago