This will be useful for writing tests which rely on shared 'global'
config between different working directories.
This also adds support for mocking 'system' (global, immutable) and
'workspace' (local, mutable). The workspace scope I think is a bit iffy
though, given how `git cl` is actually built - currently scm.GIT doesn't
really know about clone vs. workspace, and afaik no config adjustements
actually apply to the workspace scope.
Adds tests for mocked git config implementation, including bug fixes
to the current implementation revealed by the tests.
R=ayatane, yiwzhang
Change-Id: Ia56d2a81d8df6ae75d9f8d0497be0d67bdc03651
Bug: 355505750
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5759163
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Previously there was an error where you could have:
global: {"key": ["something"]}
local: {}
And called SetConfig("key", None, missing_ok=False) - this would end
up raising CalledProcessError(returncode=5) because the caching
layer would see that "key" was not missing, then call into the
storage layer which would do `git config --local ...` which was
missing "key" (and therefore fail).
R=ayatane, yiwzhang
Change-Id: Idc9dc63a1b1c0a4a203c403dbb3c2ea521099b26
Bug: 342644760
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5744333
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Allen Li <ayatane@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Otherwise pyright assumes a ton of values here are bytes
Bug: b/351071334
Change-Id: Ib360c27f6f82d35bcf885c5918b6def4ca26825f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5744594
Commit-Queue: Allen Li <ayatane@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Previously little bits of the scm.GIT were mocked (like SetConfig,
and GetConfig, but not the other config related methods).
This changes things so that the git config state is a class whose
logic is shared between prod and test, with a 'real' and 'test'
implementation which know how to load and save configuration at
a low level.
R=yiwzhang
Change-Id: I1dc11b550908862607ea539de7fa60fbce30e700
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5660891
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Allen Li <ayatane@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
For a ProvidedDiffChange, the AllFiles method naively returns all files
with rglob("*"). The returned list includes files in nested submodules.
This does not match the behavior of GitChange's AllFiles which uses
git ls-files to find all files.
Implement a new SCM that stops iterating recursively when it
sees a submodule from .gitmodules in the repo root.
Bug: b/323243527
Change-Id: I170d0f1bc4a838acea04779dee3df7fca0bce359
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5648616
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
In preparation for some Windows optimizations to how git_common calls
git it is important to use git_common more widely, specifically from
scm.py and gclient_scm.py. This change updates scm.py and gclient_scm.py
and updates the associated tests:
Test command lines used when updating the tests include:
vpython3 tests/gclient_scm_test.py ManagedGitWrapperTestCaseMock.testUpdateConflict
vpython3 tests/gclient_scm_test.py GerritChangesTest.testRecoversAfterPatchFailure
vpython3 tests/gerrit_util_test.py CookiesAuthenticatorTest.testGetGitcookiesPath
Bug: 332982922
Change-Id: I7aacb110b2888c164259815385cd77e26942adc7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5478509
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
If symbolic-ref remote HEAD is not available, we skip setting it and
query remote Git using ls-remote. Such information is not stored, and
gclient will need to repeat it on the next invocation.
Instead, we can call set-head on symbolic-ref failure. While that's
slower operation than ls-remote, it saved in internal Git database and
can be reused on next gclient invocation.
We may be okay with hardcoding 'main' as default remote branch today,
but it's possible that some projects still use old default.
R=gavinmak@google.com
Change-Id: Ic4c826b888d96e367039bfc4b9bd2ba0d8b58b52
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5492789
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts commit 3569608028.
Reason for revert: This includes a fix for crbug.com/324358728.
The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.
[1] https://codereview.chromium.org/228353003
Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
+ Change scm GetAllFiles back to returning all files
+ should be safe to do since it's only used by presubmit code
+ add ignore_submodules option to CaptureStatus
+ It's used by other parts of depot_tools code
+ Add AffectedSubmodules to return submodules only
+ AffectedFiles still only returns non-submodule files.
Bug: 1475770
Change-Id: I457ad96099fb20eff4cc1a4fc84f59e7ae707abe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5216005
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
The cache won't be used if ResolveCommit throws an exception or when
non-sha1 is used.
This optimization saves about 7% of overall gclient sync on
chromium/src.
R=gavinmak@google.com
Change-Id: I9c36d937dc7b8553818888e2e9c4eecd029a978c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5147496
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
The package is deprecated and will be removed in py 3.12.
R=ddoman@google.com
Bug: 1500415
Change-Id: Ia130c556ad3f2135fac4a9b1d9bb4ecc088cfe9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5030818
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jonathan Nieder <jrn@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Off by one - we should have 4 parts, so we need maxsplit=3.
R=aravindvasudev@google.com, jojwang
Fixed: 1474283
Change-Id: I13b568feb8ee2aa307a2267c43765c1749b77c90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4799773
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Presubmit checks expect files to be actual files. Gitlink entries are
directories (if initialized), and that breaks various existing tests.
Since we don't run presubmits from git dependencies, it's safe to ignore
any updates to gitlinks and omit them from All Files.
R=jojwang
Change-Id: I670e5caef4067a0203578e9be50c4fabf23b3153
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4795116
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
If remote HEAD returns legacy default branch, refresh HEAD by running
set-head to check if there are any changes.
R=gavinmak@google.com
Change-Id: I5891a899b512e81fccfb086a056d497c906df4b2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3152593
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Currently, gclient sync assumes the default branch is master, and
it doesn't work at all if such branch doesn't exist. This change queries
local git copy to get remote HEAD. If local git version is not
available, it queries remote git server using ls-remote.
This change requires git version 2.28 (depot_tools comes with 2.29).
R=ehmaldonado@chromium.org
Bug: 1156318
Change-Id: Id348e0f1004093f395139e8f4d62adb66b94ca9c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2628359
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
subprocess.DEVNULL was introduced in Python3 to serve same purpose
as subprocess2.VOID, so rename VOID to DEVNULL in subprocess2.
Change-Id: I6dade3306ffc3bc2441ac6083f362b099c2427e9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2587758
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This fixes presubmit checks that rely on FetchUpstreamTuple which is
used in patching.
R=ehmaldonado@chromium.org
Bug: 1143724
Change-Id: I163fa0ea6f1b356280d497196869bb206463ad4f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2515268
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
On Windows, git can be either an executable or batch script, each of
which requires carets (^) to be escaped in different ways.
This makes it hard to use `git rev-parse --verify REV^{commit}`, so we
truncate full hash revisions to achieve the same effect instead.
Change-Id: I01578620706a1bab75a2decc55d9b4f1ac3afe28
Bug: 1065307
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2127968
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This is a reland of e3a49aa405
GetRelativeRoot was fixed to use '.' instead of None
Original change's description:
> git-cl: Remove unused and duplicate functions.
>
> Remove unused functions, and use scm.GIT where possible to reduce
> duplication.
>
> Change-Id: I24f05e7b3ae04743e97b6665ee668f44d6acc294
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116938
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: I54bd1d34d3501a38bf8af2f9e70059d58fa538a5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2125293
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Remove unused functions, and use scm.GIT where possible to reduce
duplication.
Change-Id: I24f05e7b3ae04743e97b6665ee668f44d6acc294
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2116938
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Robert Iannucci
Allen Li
Gavin Mak
Bruce Dawson
Josip Sokcevic
Aravind Vasudevan
Joanna Wang
Mike Frysinger
Joanna Wang
Josip Sokcevic
Edward Lesmes
Quinten Yearsley
Edward Lesmes