This is a reland of commit 5e49eda5c4
Original change's description:
> Break make_encoded_file into two functions
>
> The python3 fallback for `make_encoded_file()` produces different results on py2 and py3 when bytes are passed. This CL makes it clear breaking it into two methods, one for text and another for bytes. This CL also adds a fallback in `download_file` for py3 to return bytes when it cannot decode the downloaded file into utf-8.
>
> Recipe-Manual-Change: chromiumos
> Change-Id: I3d313e430c852e179825bc24bf4a58ce84440b2a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3595026
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Greg Edelston <gredelston@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Recipe-Manual-Change: chromiumos
Change-Id: I2de85eca71cc45f503c25888297ba881a5c662a8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3610973
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Greg Edelston <gredelston@google.com>
M1 mac seems to have capacity to build with higher parallelism.
This also remove limit in macOS.
Bug: 1317620
Change-Id: I4460915c405cbb27ed977dcee631adb8753f2335
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3596361
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Philipp Wollermann <philwo@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This is to make the script work even when config file is broken.
Change-Id: I6c2a090b01026b5747137bcc9fdec0bd5c03f75e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3607373
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
This reverts commit 5e49eda5c4.
Reason for revert: chromiumos rollout fails
Original change's description:
> Break make_encoded_file into two functions
>
> The python3 fallback for `make_encoded_file()` produces different results on py2 and py3 when bytes are passed. This CL makes it clear breaking it into two methods, one for text and another for bytes. This CL also adds a fallback in `download_file` for py3 to return bytes when it cannot decode the downloaded file into utf-8.
>
> Recipe-Manual-Change: chromiumos
> Change-Id: I3d313e430c852e179825bc24bf4a58ce84440b2a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3595026
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Greg Edelston <gredelston@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Change-Id: Ibfb51614f9e19b575b99704aa4ca33c5525ab294
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3602168
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
The python3 fallback for `make_encoded_file()` produces different results on py2 and py3 when bytes are passed. This CL makes it clear breaking it into two methods, one for text and another for bytes. This CL also adds a fallback in `download_file` for py3 to return bytes when it cannot decode the downloaded file into utf-8.
Recipe-Manual-Change: chromiumos
Change-Id: I3d313e430c852e179825bc24bf4a58ce84440b2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3595026
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Greg Edelston <gredelston@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Header guards are checked by CheckForIncludeGuards which has broader
coverage than cpplint.py and should therefore be preferred.
Having two include-guard checkers that cover different sets of files and
use different tags to silence warnings is just confusing.
Bug: 1309977
Change-Id: Ic0d1ad610cf9c34d6777d852da2e2e22b8686552
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587725
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When running "git cl presubmit --all" some of the presubmit messages
were ending up double-spaced if the output is sent to a file and then
loaded into notepad or Visual Studio. Visual Studio would complain
about inconsistent line endings, and it turns out that this was all
caused by some lines having \r\r\n line endings - notepad and Visual
Studio interpret that as two line endings.
The problem is that the stream that WinUnicodeOutput writes to does \n
to \r\n translation. If the text being printed already has \r\n line
endings then we get doubled-up \r characters. The fix is to replace all
\r\n sequences with \n before calling write.
This reduces the line count of the output by almost 2300 lines (more
than 25% of the total) and makes it much more readable.
Bug: 1309977
Change-Id: Ie5475087badc3d3146e4f2ba41d30c9817dd375a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3589498
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
I feel like a PRESUBMIT owner now and this makes it official.
Bug: 1309977
Change-Id: I38356bc52c8d03d93c4ec42e4952b5872b31a79a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3589901
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
presubmit --all tells the presubmit system that all files are 'modified'
but ChangedContents still goes off to see what changes are present. If
most files were unchanged this was all handled perfectly, but if _all_
files were unchanged then _GitDiffCache would interpret the empty
dictionary of changes as a reason to ask git for diffs, millions of
times. This made some checks take more than 100x as long. The overall
effect on presubmit --all time is not known because I was never willing
to wait the multiple days for them to terminate.
That is, this would take many days to run:
git checkout -b empty -t origin/main
git cl presubmit --all
whereas a single-character change to any file would let this run in
about two hours.
After three weeks of working on presubmits I only hit this twice which
is why it took me so long to realize what the problem was.
Bug: 1309977
Change-Id: Ib280ea386107843b9174d835b0895316a5ed240c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3589900
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When CheckLongLines finds problems it prints the first five, but this
gives no indication as to whether there are just five, or hundreds.
This change prints the number of long lines found.
The snapshot function in PanProjectChecks prints the elapsed time for
long checks, however it has two issues that make it inconvenient. One is
that it prints the time in ms which sounds great until you get a warning
that one of the checks is taking 2041373 ms (not kidding, although now
fixed) which is tricky to read. The other problem is that it was
actually measuring CPU time, not wall-clock time. This changes it to
print the time in seconds (to 0.1 seconds) and to measure elapsed
time.
Bug: 1309977
Change-Id: I7564a8cdf7bb3349b10ebbddbfe179188d4bf309
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587726
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8816874699072805217
recipe_engine:
22391916f2
2239191 (gbeaty@chromium.org)
Prevent spurious GitFetchErrors in CommitList.from_backend.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I1f252e93fa21eff467a07dea0a74ed20458a41ab
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587404
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8816875963485349249
recipe_engine:
e783f88f68
e783f88 (chanli@chromium.org)
[recipe-py] (2 attempt) buildbucket: set `luci.buildbucket.pare...
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I277b6a32a528dc3d1fd8fe8237f6ada9c61c8f52
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587403
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8816888547939077841
recipe_engine:
d63eb67ccf
d63eb67 (chanli@chromium.org)
Revert "[recipe-py] buildbucket: set `luci.buildbucket.parent_t...
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: If35beef29b7264610b3414c8c3af7afc77b5ba62
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587286
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8816954607693384897
recipe_engine:
048f6f2fbb
048f6f2 (chanli@chromium.org)
[recipe-py] buildbucket: set `luci.buildbucket.parent_tracking`...
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I8cdd9dda8c4160dfb399b59b3d41b12d216bc8be
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3581205
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
After optimizing CheckForIncludeGuards the CheckLicense presubmit was
the second slowest presubmit, taking about 35 minutes when run as part
of "git cl presubmit --all". The cause of its slowness was initially
non-obvious because it runs quite quickly on most files, however
analysis showed that it could take 50+ seconds on some files. The files
that it is slow on are those that lack a good license header, meaning
that the regex match has to painstakingly scan the entire file.
The optimization in this change is to recognize that there is a simple
non-regex line that appears in all valid license headers, regardless of
variants. If that line is absent then there is, necessarily, no valid
license header, and searching for a line of text is something that
Python can do extremely quickly.
This change drops the CheckLicense time from about 35 minutes to about
32 seconds.
Trivia: _CommonChecks in third_party/blink/PRESUBMIT.py passes .* as the
license, so I added an early-out for that to avoid pointlessly scanning
those files.
Bug: 1309977
Change-Id: Ic2e56079675c2c5a2643d20dd492b1cc52e4ead2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3584882
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This is a reland of commit 053817260a
The change was not related to the errors that it was reverted for, so no
changes are necessary.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: I80efeee8a2b951df43b00d89bb596dc2cf2fcec7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3582182
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
This reverts commit 053817260a.
Reason for revert: we reverted got_revision change, and this change
itself may be causing some other failues.
Original change's description:
> Set a default got_revision property in the bot_update json output.
>
> The bot_update.py script sets a got_revision property by default even if
> it is not present in the reverse revision mapping. This results in an
> uncaught exception when set_output_commit is set to True if got_revision
> isn't present in the reverse revision mapping, but it isn't caught until
> production because the test API doesn't match that behavior. This change
> updates the test API method to match the behavior of the script.
>
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: infra
> Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Garrett Beaty <gbeaty@google.com>
> Auto-Submit: Garrett Beaty <gbeaty@google.com>
Change-Id: Ie975dfe39e7b8997336761e12f53f3046359d825
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3578820
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
If a user has git for x64, `git cl` results in downloading x64 cipd
packages. Other operations, such as gclient or git-cl will result in
downloading arm64. This flip-flopping between architectures causes
unexpected behavior:
(1) slows down commands since packages are wiped and new ones are
downloaded.
(2) long running processes, such as goma may crash.
This change adds additional check for all macs running on x86 to check
if is running under Rosetta and sets ARCH=arm64 if true.
R=bryner@google.com, masonf@chromium.org
Bug: 1311733
Change-Id: I65b127467d5c47d5bf07952d2ecdcb93630c7c87
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3573068
Reviewed-by: Brian Ryner <bryner@google.com>
Reviewed-by: Mason Freed <masonf@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
The bot_update.py script sets a got_revision property by default even if
it is not present in the reverse revision mapping. This results in an
uncaught exception when set_output_commit is set to True if got_revision
isn't present in the reverse revision mapping, but it isn't caught until
production because the test API doesn't match that behavior. This change
updates the test API method to match the behavior of the script.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Change-Id: Ideefa9d77d2a816ae66a2bb52737264ed3f5bcee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3575361
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
Auto-Submit: Garrett Beaty <gbeaty@google.com>
swift format is only available on mac hosts.
Swift formatting is disabled by default. It will be
enabled by default on mac host in another CL.
Change-Id: Id603be203edd44b6419d545027c249111abe1baf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3550197
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Olivier Robin <olivierrobin@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
The build that created this CL was
https://ci.chromium.org/b/8817601997489655649
recipe_engine:
a6fd0d2d3c
a6fd0d2 (chanli@chromium.org)
[time] set the step to canceled if GLOBAL_SHUTDOWN is triggered
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Ignore-Freeze: Autoroller
Bugdroid-Send-Email: False
Change-Id: I1a2adbaa82f833cb68fd730c213070c6c41adda2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3574982
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
I regularly run into this error for local branches corresponding to
merged CLs. Before, I had to manually type `git rebase --abort`
before copy-pasting the `git config branch.[...].dormant true` line
into my terminal. With this change, I can just copy-paste once to
execute both steps, then press up+up+enter to retry.
As a side benefit, Git newcomers won't have to look up how to abort
a rebase.
Anyone preferring the previous workflow can still elect to copy
just the `git config` line.
Bug: none
Change-Id: I516d4ed12c5111936cc65460ce119a43e5f86a8b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3570882
Auto-Submit: Chloe Pelling <cpelling@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
vpython3.bat adds an entry to the path. Unfortunately that modification
persists after the batch file returns. Repeated running of vpython3.bat
will eventually grow the path so long that the command to update the
path exceeds the 8191 character cmd.exe command-length limit. This can
most easily be seen by repeatedly running this command:
vpython3 -c "print('Hello world!')" && set path | wc
After about 35 invocations (dependent on exact system setup) this
happens:
>vpython3 -c "print('Hello world!')" && set path | wc
The input line is too long.
The syntax of the command is incorrect.
This batch also (but less critically) leaks the PYTHON3_BIN_RELDIR
environment variable.
The fix is to add a setlocal command to keep environment changes local.
Bug: 1003139
Change-Id: I11a31e1013017da702299bfe8eb4888985c228f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3566378
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Brian Ryner <bryner@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Aravind Vasudevan
Junji Watanabe
Takuto Ikuta
Bruce Dawson
recipe-roller
Ravi Mistry
Ryan Heise
David Dorwin
Fumitoshi Ukai
Garrett Beaty
Eduardo Yap
Josip Sokcevic
Jack Neus
Yannic Bonenberger
Olivier Robin
Brian Ryner
Chloe Pelling