Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
No script can run on py2 (due to print syntax change), so those
checks are useless (will never be printed).
Change-Id: I9cc7bc8b2665dd09cc7f8939ba255956f96af88a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5262958
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts commit 92b780b0bd.
Reason for revert: mac arm experiencing hang
Original change's description:
> [gclient] Enable parallel sync on ARM Mac by default
>
> I realized that gclient sync doesn't use all the cores on ARM Mac.
> This CL changes the default for ARM Mac to use all the available cores.
>
> Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
> Commit-Queue: Junji Watanabe <jwata@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Junji Watanabe <jwata@google.com>
Bug: 1427050
Change-Id: I512d1d1ef0034f12fc9054c5234ae688ec281857
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5259766
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
I realized that gclient sync doesn't use all the cores on ARM Mac.
This CL changes the default for ARM Mac to use all the available cores.
Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Change-Id: Ic2d3ea0dbaad27aad13dcfc6cc3c4f5c1fd8c545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5055942
Reviewed-by: Scott Lee <ddoman@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit 8672d51b79.
Reason for revert: crrev.com/c/5018522 has landed, so this is no longer required.
Original change's description:
> gclient: Add skip-git experiment.
>
> This experiment causes all dependencies of type 'git' to be skipped for
> all commands. This can be useful if you need to build Chromium in a
> working tree that is guaranteed to contain a consistent, pre-synced
> snapshot of the code that's managed by Git, but still need to install
> cipd dependencies and run the hooks to download the remaining
> dependencies.
>
> Bug: b/310485239
> Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Philipp Wollermann <philwo@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bug: b/310485239
Change-Id: If1fc93d493daa9515ddfb568aecf4374deb6ec03
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035260
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This experiment causes all dependencies of type 'git' to be skipped for
all commands. This can be useful if you need to build Chromium in a
working tree that is guaranteed to contain a consistent, pre-synced
snapshot of the code that's managed by Git, but still need to install
cipd dependencies and run the hooks to download the remaining
dependencies.
Bug: b/310485239
Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
.git suffix is not well supported by the systems that utilize
submodules. Since some tools expect DEPS to have .git suffix, it's not
possible to simply update DEPS without breaking the tools.
This patch updates `gclient gitmodules` to omit .git suffix and trailing
slashes. However, if `.gitmodule` already exists, we are expecting the
tool to generate the same result if DEPS didn't change.
R=aravindvasudev@google.com
Bug: b/303296048
Change-Id: I30497b9d5f4e9681e16b4a913151de0f4a7c9545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5008942
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Change-Id: I5d1c43546bb4ac57b9bb57f47e9ef04f1b19bc0c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4881087
Auto-Submit: Philipp Wollermann <philwo@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Philipp Wollermann <philwo@google.com>
With submodules, users can accidentally stage and commit gitlink
changes. Add a new gclient command to install a pre-commit hook to
automatically drop gitlink changes that don't correspond to a DEPS
change.
Dropping gitlinks can be bypassed by setting
SKIP_GITLINK_PRECOMMIT=1.
Bug: 1481266
Change-Id: Idd8b273e7d8e37d52627964e8ed6004d068b6b7a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4863221
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
python3 is the only supported version of python in depot_tools.
Changes include:
* updating python version requirements
* removing third_party/six
Bug: 1475402
Change-Id: Ib86231413fcac494ceb27d1783e53b9d86533c26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4845292
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
If git submodules and DEPS ever go out of sync, there are two different
behaviors in gclient:
* gclient gitmodules needs to read DEPS file and generate submodules
* all other gclient operations prefer content of git submodules.
Since we enforce SYNC to always match their content via presubmit
checks, it's reasonable to expect they will only diverge on development
environment. User is more likely to update DEPS file by hand than to
gitlink.
If user updates a gitlink accidentally, gclient sync won't checkout to
content of DEPS which then will require user to take additional steps to
recover.
This change uses DEPS to be source of truth for git dependencies if
git_dependencies == SYNC.
R=aravindvasudev@google.com, jojwang@google.com
Bug: 1476108
Change-Id: I5b22095b5c9ec764ea6bf07310a0e5bba4a71aad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4820459
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Removing some deadcode from depot_tools scripts. More follow-up CLs will
be made after this to remove more dead code.
R=sokcevic,gavinmak
Change-Id: Iad20e9e1737e49c1370248d32c6317cf2be63de4
Bug: 1475776
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4811216
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Use universal newlines when writing to gitmodules. Also, split
update-index command so we don't hit Windows limit for process lenght.
R=jojwang
Bug: 1475328
Change-Id: Ia9bba4620a276c31081a9773bac8f6bd309bb86e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4808507
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
When `gclient setdep` has `--deps-file` arg provided, use its basedir to
run submodule commands. And fail gracefully when thsoe commands throw
exceptions.
Change-Id: I94192b20fb5c34729c75404a34762546f6117689
Bug: 1472449
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4775173
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
The filepath already has cwd appended so the command doesn't need cwd.
Change-Id: I20c6fea27aee963e90cdb8a15cb087771dafa0fa
Bug: 1472143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4771942
Commit-Queue: Scott Lee <ddoman@chromium.org>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Also add oneline status on gclient gitmodules.
R=jojwang@google.com
Change-Id: I05c9f856ce6fd1c3ebf1dc7da672d25196a4cb67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4771975
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
The `--format` option for `git ls-tree` was introduced in git v2.36 and
hence gclient breaks for lower git versions.
Change-Id: I02750904aa78bb5fd5f3f57dc2ad42e355fa4a42
Bug: 1471241
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4765682
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
The ParseGitModules() uses `git submodule status` result to get the
commit hashes for the submodules. It assumes the command's result to not
have a `git describe` at the end of each line, and hence only works in
certain scenarios. This change fixes this by updating the parser.
Change-Id: I8f4e3fae6562dafd0e127255a8f26e7336a75683
Bug: 1471241
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4763348
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This makes hooks path behavior match deps path behavior.
Bug: 1468658
Change-Id: Ib6685cb52679249b389599ada549e2c4410e812c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4729084
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
We expect gclient to place git submodules in the same directory where
.gitmodules is, and without any modification.
Prior to this patch, gclient reads paths and use them as-is. So, if a
project ./foo/ has submodule 'bar', the git repository will be placed in
./bar instead of ./foo/bar iff use_relative_paths is set to false.
Bug: 1429149
Change-Id: I4f99bb11a20d9779c1226926187ebfd8f5d829a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4680282
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This is a reland of commit e27d7c3816
Original change's description:
> Drop support for py2 presubmit hook win mapping
>
> We no longer support py2 so we shouldn't have a special handling for py2
> hooks.
>
> R=gavinmak@google.com
>
> Change-Id: I038f462a4991d17bedb3e02104374b3ad1ffa5ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4541664
> Commit-Queue: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Change-Id: I190fb7c1ee4e76adfe74196c1ed579a424314a02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4669591
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This change adds `git_dependencies` flag to the DEPS. This will be used to track the submodule migration state.
Design Doc: go/depot-tools-on-submodules
Change-Id: I547ad8ebbce2535960d4b6e00c0b14c00118f544
Bug: 1429149
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4591777
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Michael Ershov
Stephanie Kim
Gaby Baghdadi
Aravind Vasudevan
Josip Sokcevic
Junji Watanabe
Alexei Svitkine
Gavin Mak
Takuto Ikuta
Philipp Wollermann
Philipp Wollermann
Joanna Wang
Yiwei Zhang
Mike Frysinger