The current owners systems scales badly on the number of files in a
patch, on the number of OWNERS files and on the number of "set noparent"
rules. This patch addresses all three of those.
If you have a patch with many (thousands) of files, you run into a lot
of repeated work because the parent directories are visited for every
file. By using a cache or remembering what directories have been
visited, a lot of work can be avoided.
If you have many OWNERS files or "set noparent" rules in scope of a
patch, owners and stop_looking lookups became slow. By splitting the
rules by longest glob-free directory path, you get much smaller lists
to process, which makes the code much faster.
(Test case with "all the files" went from >15 minutes to 30 seconds).
This saves about 2.5 second for git cl upload with a patch of
350 files spread over the tree.
The addition of many noparent rules in the Chromium tree might have
been the major reason this is needed, but there are also more files to
change, allowing larger patches.
Bug: 920591
Change-Id: If21178746ca9b88c2c07d265bd583b556d1734b7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1404172
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Daniel Bratell <bratell@opera.com>
Commit-Queue: Daniel Bratell <bratell@opera.com>
When NINJA_SUMMARIZE_BUILD=1 then autoninja prints a summary of build
performance at the end. This change makes it so that ninja prints more
detailed build statistics during the build. In particular, slow process
creation is a common but difficult to see bottleneck. This change makes
it so that if NINJA_SUMMARIZE_BUILD=1 then the number of running build
processes will be displayed.
Bug: 787983
Change-Id: Ic0907e23e7f762e23e4795059b37085e0cb8c4ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1582802
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This change should hopefully enable the openscreen repo to use the
run_presubmit recipe.
Bug: openscreen:43
Change-Id: I672cddfeddd09ccc25ca851c0b613293191f17f9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1574189
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Bash and its descendants allow having $PATH entries containing literal tildes,
like:
...:~/depot_tools:...
If the user has their shell configured this way, their shell will be able to
invoke `fetch` and other depot_tools commands, but depot_tools will not in turn
be able to invoke subcommands via popen because popen does not have this
bash-specific behavior when not in shell mode.
This change has gclient detect this configuration problem and error out with a
descriptive error message. Not doing so leads to a puzzling condition where
gclient complains about being unable to find a program that (from the user's
perspective) is in their $PATH and invokable interactively.
It also has 'fetch' suppress its own failure stack trace if it runs gclient and
gclient fails; the 'fetch' stack trace pushes the messages from gclient (which
are actually diagnostic) off the screen.
Bug: chromium:952865
Change-Id: Ibba4d2fccee405aa68392ce141493f1de21ec018
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1573204
Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
A previous CL replaced "basestring" with "str", because basestring
does not exist in Python 3. However, this broke Python 2's ability
to interoperate with unicode strings. This CL introduces a workaround
(defining basestring to be equivalent to string, if it doesn't exist
already), and restores the references to basestring. This workaround
can be fixed when we're 100% on Python 3.
It also undoes some unnecessary and harder-to-read formatting changes.
Bug: 942522
Change-Id: I4a31ee46dc048134c2e4832b6c44ea00ce341899
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1572441
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
- It uses a different commit subject than recipe roller.
- Switch from looking up for magic strings to use a regexp instead. Document the
source of each string, to help keep up when the source diverge.
Tested manually.
Change-Id: I76fc6b6692508c615a0e76c4b6b403f8c513dd31
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1564938
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Restore previous behavior and keep emitting a warning when an existing
gclient entry is not longer part of the client. Additionally, suggest
the usage of `gclient sync -D` to automatically remove them.
Change-Id: I62d4662ae4d0886d340230019419b68debffc5ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1483031
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Henrique Ferreiro <hferreiro@igalia.com>
We should only be maintaining one cache bundle per repo, but it turns
out that we've had two in the past due to GoB supporting two different
paths to the repo, and users were getting stale bundles as a result.
This CL fixes things so that we should only get a single bundle per
repo.
Bug: 935084
Change-Id: I0d6713280a2abbc20e35ff87e7be115870dd5140
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1566431
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Hooks for Electron output UTF-8 characters.
Example error for when "✔" is output:
File "/home/markus/depot_tools/metrics.py", line 266, in print_notice_and_exit
yield
File "/home/markus/depot_tools/gclient.py", line 3112, in <module>
sys.exit(main(sys.argv[1:]))
File "/home/markus/depot_tools/gclient.py", line 3098, in main
return dispatcher.execute(OptionParser(), argv)
File "/home/markus/depot_tools/subcommand.py", line 252, in execute
return command(parser, args[1:])
File "/home/markus/depot_tools/gclient.py", line 2677, in CMDsync
ret = client.RunOnDeps('update', args)
File "/home/markus/depot_tools/gclient.py", line 1746, in RunOnDeps
self.RunHooksRecursively(self._options, pm)
File "/home/markus/depot_tools/gclient.py", line 1052, in RunHooksRecursively
hook.run()
File "/home/markus/depot_tools/gclient.py", line 245, in run
cmd, cwd=self.effective_cwd, always=self._verbose)
File "/home/markus/depot_tools/gclient_utils.py", line 344, in CheckCallAndFilterAndHeader
return CheckCallAndFilter(args, **kwargs)
File "/home/markus/depot_tools/gclient_utils.py", line 576, in CheckCallAndFilter
stdout.write(in_byte.decode())
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 0: unexpected end of data
This issue was introduced in CL:1524583.
Bug: 942522
Change-Id: I3c4355b925b34398c800d142f942531a829e0297
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1541334
Auto-Submit: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Raul Tambre <raul@tambre.ee>
This reverts commit 9198ef8ede.
Reason for revert: Spec revert: appears to be causing large number of chromium CQ failures on patch application.
Original change's description:
> [bot_update] Default to refs/heads/master
>
> When revision is not specified, currently HEAD is checked out.
> Change the default to refs/heads/master
>
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Bug: 940214
> Change-Id: I8be59095d90447ea898d5247a6433f0734f5e63b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1554862
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
TBR=nodir@chromium.org,tandrii@chromium.org,jbudorick@chromium.org
Change-Id: Icafd817d8f044a4347342bd85dc97d193329a704
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 940214
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1555514
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
When revision is not specified, currently HEAD is checked out.
Change the default to refs/heads/master
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 940214
Change-Id: I8be59095d90447ea898d5247a6433f0734f5e63b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1554862
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This is a reland of 3c830222ae with a fix.
Previous CL relied on summary data of CQ votes provided by Gerrit,
which apparently isn't always present in the way I expected.
This resulted in "git cl status" returning "lgtm" instead of "commit"
on Skia recipe roll CLs, which resulted in recipe roller abandoning CLs,
and creating new ones (there might be a bug in recipe roller, too).
The fix is to iterate over all votes on CQ label and finding the highest
score, and then deciding on the state.
Original change's description:
> git cl status: differentiate "dry-run" from generic "commit".
>
> CQ vote now before
> +2 "commit" "commit"
> +1 "dry-run" "commit"
>
> R=ehmaldonado
>
> Tested: manual "git cl status" run locally.
> Change-Id: I342a3ac0830a67bf0e722c737ef00460a0df4b98
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1548287
> Auto-Submit: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: If9af5a2bf8f21d203b7a62740a27916eca0838ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1549640
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This should allow for any recipe that uses the ANGLE repo to run
gclient sync with the right folder structure. Currently gclient sync
will check out third_party directories to the wrong folder.
Bug: angleproject:3288
Change-Id: Ia97592fdc0c9f3ec70c2d4a05d9934a6bc996507
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1551019
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Add set_output_commit param to bot_update.ensure_checkout, false by default.
If true, call api.buildbucket.set_output_gitiles_commit to set the output commit
of the build. This generally supersedes got_revision and got_revision_cp.
Bug: 940214
Change-Id: I5ffc790a6b12c8bf4a56469b3ecc567141c0d735
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1548256
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Karen Qian
recipe-roller
Daniel Bratell
Bruce Dawson
Vadim Shtayura
btolsch
Elly Fong-Jones
Aaron Gable
Marc-Antoine Ruel
Henrique Ferreiro
Andrii Shyshkalov
Dirk Pranke
Nodir Turakulov
John Budorick
Raul Tambre
Derek Schuff
Jamie Madill
Edward Lemur