There are 3 layers modified, starting from the bottom up:
1. git_cache.py populate
Now takes a --no-fetch-tags option. If specified, the cache will not
fetch updated tags from the server by passing --no-tags to git fetch.
This prevents the server from sending all the tag refs. In chromium
this prevents significant time bottlenecks dealing with 10k+ tags.
2. bot_update.py
bot_update has to deal with multiple git repos, it has the root repo
that is checked out through git-cache, as well as repos synched via
DEPS and gclient.
The script now takes a --no_fetch_tags option. If specified,
the git-cache populate call will include --no-fetch-tags. Otherwise, it
won't. So this controls (for chromium) if fetches to the src.git server
are passed --no-tags.
3. bot_update/api.py
This is the entry point for recipes and also has to deal with multiple
git repos. The behaviour at this point is not changed from the default.
A |no_fetch_tags| parameter is added to ensure_checkout() that defaults
to False.
This CL is a refactor with no intended behavior change.
The next step will be to change the chromium build repo to pass True
for ensure_checkout(no_fetch_tags) on chromium trybots.
This represents solution #2 in
https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit#
Bug: 1019824
Change-Id: I935430603299daa9e301a95a5184c0ce486fd912
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
This reverts commit c87ed60623.
Reason for revert:
Causes a regressions for CL descriptions on Windows
Original change's description:
> git-cl: Fix some python3 compatibility errors.
>
> Also, fix bug in git cl status where the 'updated' field was used to compare messages, even though
> it doesn't exist (see [1]). This CL modifies it to use 'date', which does exist.
>
> [1] https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#change-message-info
>
> Bug: 1002209
> Change-Id: I5a5e1193b8502c3ad35d94808ea178cad7f44ac6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891259
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Anthony Polito <apolito@google.com>
TBR=ehmaldonado@chromium.org,apolito@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 1002209
Change-Id: I004f202b12c6b99cb6b24cb12a14fba7898569e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1898547
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Sorting owners by score is non-deterministic, so sort by
name too.
Bug: 1009814
Change-Id: I93a9370a5852634b4bbb2768b9ebbb5b866348e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891665
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
This makes no-rebase-patch-ref correctly populate `file_list` and work with
`git diff --cached` (i.e. they will show only the files affected by the
patchset).
Previously gclient would `reset --soft` to master. Because we didn't rebase
on top of master with `no-rebase-patch-ref`, it has roughly nothing to do
with the patchref we just checked out.
R=gbeaty@chromium.org, ltina@google.com, tandrii@chromium.org
Context:
Tricium recipes sometimes need to use no-rebase-patch-ref in order to get
accurate linenumbers in the diff (i.e. so that the tricium comments will
actually line up correctly with the patchset the user uploaded); Tricium
recipes also need to accurately get the diff file list in this mode.
Change-Id: I5f3c95cd4958cf407a83b96c238b8c55c452ac81
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1895041
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
The finder.owners_queue contains a set of owners but they appear in an
arbitrary order on each invocation of the test. The current method of
assertEqual() with a list fails randomly because of the ordering being
different.
Example: https://ci.chromium.org/p/infra/builders/try/Depot%20Tools%20Presubmit%20(win)/b8897976148721436512
This changes comparisons of owners_queue to use an assertEqualSets()
which does a comparison that is order-independent.
Change-Id: Ia827795af283dadeb56f76ae0cf14fa4ad7e9eeb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1895699
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Garrett Beaty <gbeaty@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
The problems on crbug.com/1017367 should have been fixed by crrev.com/c/1880013
Bug: 1009814
Change-Id: Ia98304ca983b4e7e293fedb9df1fee589de58ba9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1887714
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
It's not used and uses basestring, which is not supported in Python 3.
Bug: 1002209
Change-Id: I9df900076978636c4a5c75e84432a8cc02d46909
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891654
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
basestring doesn't exist in Python 3. scm.Capture status doesn't need to support strings either.
Bug: 1002209
Change-Id: Ie9edfdfd09b789bfaace038a6ad8f3a7da344bc4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1890925
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 512f92b731.
Reason for revert: bot_update is having infra failure on ci bots.
For example https://ci.chromium.org/p/chromium/builders/ci/Closure%20Compilation%20Linux/249684.
Original change's description:
> Fetch the input commit ref when both ref and id are specified.
>
> This causes --refs refs/head/master to be added to bot_update
> calls when the input build message contains a commit id and the
> refs/head/master ref. This is an effective no-op.
>
> Bug: 922150
> Change-Id: I119d9b3cede727713a0b66bff83b5ee4e1848790
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1884457
> Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
> Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
TBR=tandrii@google.com,gbeaty@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 922150
Change-Id: Ie9275c21827688f59449f052c768452a8652e139
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891000
Commit-Queue: Haiyang Pan <hypan@google.com>
Reviewed-by: Haiyang Pan <hypan@google.com>
This reverts commit 1e4dbf3f64.
Reason for revert: unlikely but possible cause of http://shortn/_sgL4PqICVB
Original change's description:
> Add more git tracing.
>
> This CL adds more perf tracing to `git fetch` from bot_update.py and
> git_cache.py
>
> Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: infra
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Auto-Submit: Erik Chen <erikchen@chromium.org>
TBR=danakj@chromium.org,tandrii@google.com,erikchen@chromium.org
Change-Id: I03b33bf1532212f83a6b89b1de32fafb6b8aafc4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891001
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Checking for LUCI_CONTEXT is not enough. To use local auth, the
stored config must be valid and have a default account id.
Bug: 1018069
Change-Id: I5bbe880cc217a06ae0e1ae8d7e00ef09502af6a9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888211
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This CL adds more perf tracing to `git fetch` from bot_update.py and
git_cache.py
Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Auto-Submit: Erik Chen <erikchen@chromium.org>
This causes --refs refs/head/master to be added to bot_update
calls when the input build message contains a commit id and the
refs/head/master ref. This is an effective no-op.
Bug: 922150
Change-Id: I119d9b3cede727713a0b66bff83b5ee4e1848790
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1884457
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
This is a reland of 5bfa3ae88d
Replace cStringIO with StringIO and add tests.
Original change's description:
> gerrit_util: Refactor ReadHttpResponse and add more tests.
>
> Bug: 1016601
> Change-Id: Ie6afc5b1ea29888b0bf40bdb39b2b492d2d0494c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880014
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Bug: 1016601
Change-Id: I0c83a83202169b6a1acc60bdf6f4cd00eac6e2a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1884461
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 5bfa3ae88d.
Reason for revert:
Fails when fetching status for cl 1708084
Original change's description:
> gerrit_util: Refactor ReadHttpResponse and add more tests.
>
> Bug: 1016601
> Change-Id: Ie6afc5b1ea29888b0bf40bdb39b2b492d2d0494c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880014
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
TBR=ehmaldonado@chromium.org,apolito@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 1016601
Change-Id: I84bc378ed5f58e911e0900b1a5dbea70dc06ade1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1883677
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Previously, it returned bytes if the file did not contain valid Unicode data,
and a Unicode string otherwise.
It is confusing to reason about such a function, and no current caller needs
bytes data AFAICT, so make FileRead always return Unicode strings.
Bug: 1009814
Change-Id: I89dd1935e5d4fcaf9af71585b85bda6c47695950
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880013
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This fixes an error on some Windows installations:
```
fetch webrtc
Traceback (most recent call last):
File "googlesource.com\depot_tools\bootstrap\bootstrap.py", line 365, in <module>
sys.exit(main(sys.argv[1:]))
File "googlesource.com\depot_tools\bootstrap\bootstrap.py", line 324, in main
git_postprocess(template, os.path.join(bootstrap_dir, 'git'))
File "googlesource.com\depot_tools\bootstrap\bootstrap.py", line 264, in git_postprocess
maybe_copy(
File "googlesource.com\depot_tools\bootstrap\bootstrap.py", line 108, in maybe_copy
content = fd.read()
File "googlesource.com\depot_tools\bootstrap-3_8_0b1_chromium_1_bin\python3\bin\lib\encodings\cp1251.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 18734: character maps to <undefined>
```
Bug: None
Change-Id: I43cf7b51879ac9a66c33566536dcdcb4c93e0fc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1881227
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Most notable public API change is that '../' is no longer allowed in
lucicfg.emit(dest=...).
R=tandrii@chromium.org
Change-Id: Iff162c6e1cbd0c7fcbbafc8b2f56eb827660aa93
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1880301
Auto-Submit: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Lets the client repository make 'git cl format' use the
'--full' option by default. This solves issues when using less
common clang-format options that don't behave well with small
diffs. For example, AlignConsecutiveAssignments.
This is a replacement for the 'diff-lines-of-context' setting.
Bug: angleproject:4003
Change-Id: I81dc3b4992a7420e7235da88ec78e51ec4c0d24f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1879148
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Exceptions other than OSError are not surfaced.
This caused errors like this to be printed, but not block presubmit,
allowing bugs to sneak in.
Exception in thread Thread-8:
Traceback (most recent call last):
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\presubmit_support.py", line 199, in _WorkerFn
result = self.CallCommand(test)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\presubmit_support.py", line 170, in CallCommand
p = subprocess.Popen(cmd, **test.kwargs)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\subprocess2.py", line 143, in __init__
super(Popen, self).__init__(args, **kwargs)
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\subprocess.py", line 640, in _execute_child
startupinfo)
TypeError: environment can only contain strings
https://logs.chromium.org/logs/infra/buildbucket/cr-buildbucket.appspot.com/8898840708364523888/+/steps/presubmit/0/stdout
Change-Id: I34e65d8c0050eed7ed26fd782e0a5dc8616f30f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1877051
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
This reverts commit 812ac228e3.
Reason for revert: The fix was not sufficient to solve the git cl
format difference between 'git cl format' and 'git cl format --full'.
Looks like we'll need a different solution.
Original change's description:
> git-cl: Add diff lines of context setting.
>
> Clients using some Clang-Format options can run into bugs when using
> 0 lines of context. AlignConsecutiveAssignments in particular is picky
> and behaves differently with 0 lines of context.
>
> This behaviour lead to 'git cl format' behaving differently from
> 'git cl format --full'. That could also break presubmit format checks.
> Using >0 lines of context for 'git diff' fixes the inconsistency.
>
> Add a setting 'DIFF_LINES_OF_CONTEXT' that can be controlled via
> codereview.settings. Defaults to zero to preserve the old behaviour.
> The setting allows the client to control the number of lines of context
> to use when running 'git cl format'.
>
> Bug: angleproject:4003
> Change-Id: Ied2ebf23df4c41ba19bfbd5b8ddf526b56a20b31
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1864309
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=agable@chromium.org,ehmaldonado@chromium.org,jmadill@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:4003
Change-Id: Ib6b70969509a1e5446922d488f11d13a0c59c536
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1877003
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
The support/ dir has only ever been used to host a single CrOS file.
We can move that to `cros` (which is the primary tool in the CrOS
world), and have the few other wrapped programs point to that.
Bug: None
Change-Id: I3ba3cc7375d357d62fb464e1b6dc37e73bc83cb5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1876639
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
recipe-roller
Vadim Shtayura
danakj
Edward Lesmes
Edward Lesmes
Robert Iannucci
Haiyang Pan
John Budorick
Fumitoshi Ukai
Erik Chen
Garrett Beaty
Yura Yaroshevich
Jamie Madill
Takuto Ikuta
Ben Pastene
Mike Frysinger
Andrii Shyshkalov