This is so that the third_party directories can gitignore the
downloaded tarfile. Currently theyre ignoring on *tar.gz
Bug: b/324418194
Change-Id: Id4aeeec366de3a0cec410de6ddd63369a3a4680a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454165
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The sysroot tarfile's getnames() and getmembers() functions returns
paths that begin with './' and a '.'. Removing the
leading './' helps determine which of them are probable top level
directories. Also ignore the '.'
Bug: b/324418194
Change-Id: I30667b685ee85e55f0146a743701aafb4094c928
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5437002
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
When installing a gcs dep entry, record the getnames() output to
.<object_name>_content_names. When updating the object(s) of a gcs
entry, it'll clear all files related to that gcs entry including:
.<object_name>_content_names and the paths inside it,
.<object_name>_hash, and .<object_name>_is_first_class_gcs
Bug: b/324418194
Change-Id: Ieb56623ce929b715ed103af9560efcf66b46a9c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454646
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Now that multiple objects can share a directory, when objects are
removed, the directory should also remove the extracted contents
from that specific object. Since those exact contents are unknown, the
whole directory will be cleared.
If an entire GCS dep is added or removed, the corresponding directory
path will be cleared as well.
.gcs_entries holds a record of which GCS deps and objects
have been downloaded, per checkout. Example:
```
{
"src": {
"src/third_party/llvm-build/Release+Asserts": [
"Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz",
"Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz"
],
"src/third_party/node/linux": [
"46795170ff5df9831955f163f6966abde581c8af"
]
}
}
```
Bug: b/324418194
Change-Id: Icac113572523b61c83450880615418bf7df8bba7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5407888
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This script allows users to upload one file or one or more compressed
directories to GS. It will also output the DEPS blob with the required
sha256sum and size_bytes.
Verified script locally.
Bug: b/328065301
Change-Id: Ic8e894917eb4efde4f287ffb6869a22be1e89bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381863
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This utilizes the same tarfile validation as
download_from_google_storage, but checks that the links and contents
are inside the tarfile's multiple top level directories instead of
just one directory inside the tarfile.
Bug: b/324418194
Change-Id: I73505c72084d54ca8088213dbc40b371675f8dc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381117
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
GCS deps will require a size_bytes field of type `int`. If gclient
sees that the actual byte size is different than the noted size, it
will raise an exception.
Bug: b/328065441
Change-Id: If61496eae39cd372966d7e64ba4a84759708c60b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5367540
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
No script can run on py2 (due to print syntax change), so those
checks are useless (will never be printed).
Change-Id: I9cc7bc8b2665dd09cc7f8939ba255956f96af88a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5262958
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts commit 92b780b0bd.
Reason for revert: mac arm experiencing hang
Original change's description:
> [gclient] Enable parallel sync on ARM Mac by default
>
> I realized that gclient sync doesn't use all the cores on ARM Mac.
> This CL changes the default for ARM Mac to use all the available cores.
>
> Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
> Commit-Queue: Junji Watanabe <jwata@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Junji Watanabe <jwata@google.com>
Bug: 1427050
Change-Id: I512d1d1ef0034f12fc9054c5234ae688ec281857
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5259766
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
I realized that gclient sync doesn't use all the cores on ARM Mac.
This CL changes the default for ARM Mac to use all the available cores.
Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Change-Id: Ic2d3ea0dbaad27aad13dcfc6cc3c4f5c1fd8c545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5055942
Reviewed-by: Scott Lee <ddoman@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit 8672d51b79.
Reason for revert: crrev.com/c/5018522 has landed, so this is no longer required.
Original change's description:
> gclient: Add skip-git experiment.
>
> This experiment causes all dependencies of type 'git' to be skipped for
> all commands. This can be useful if you need to build Chromium in a
> working tree that is guaranteed to contain a consistent, pre-synced
> snapshot of the code that's managed by Git, but still need to install
> cipd dependencies and run the hooks to download the remaining
> dependencies.
>
> Bug: b/310485239
> Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Philipp Wollermann <philwo@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bug: b/310485239
Change-Id: If1fc93d493daa9515ddfb568aecf4374deb6ec03
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035260
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This experiment causes all dependencies of type 'git' to be skipped for
all commands. This can be useful if you need to build Chromium in a
working tree that is guaranteed to contain a consistent, pre-synced
snapshot of the code that's managed by Git, but still need to install
cipd dependencies and run the hooks to download the remaining
dependencies.
Bug: b/310485239
Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
.git suffix is not well supported by the systems that utilize
submodules. Since some tools expect DEPS to have .git suffix, it's not
possible to simply update DEPS without breaking the tools.
This patch updates `gclient gitmodules` to omit .git suffix and trailing
slashes. However, if `.gitmodule` already exists, we are expecting the
tool to generate the same result if DEPS didn't change.
R=aravindvasudev@google.com
Bug: b/303296048
Change-Id: I30497b9d5f4e9681e16b4a913151de0f4a7c9545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5008942
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Change-Id: I5d1c43546bb4ac57b9bb57f47e9ef04f1b19bc0c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4881087
Auto-Submit: Philipp Wollermann <philwo@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Philipp Wollermann <philwo@google.com>
With submodules, users can accidentally stage and commit gitlink
changes. Add a new gclient command to install a pre-commit hook to
automatically drop gitlink changes that don't correspond to a DEPS
change.
Dropping gitlinks can be bypassed by setting
SKIP_GITLINK_PRECOMMIT=1.
Bug: 1481266
Change-Id: Idd8b273e7d8e37d52627964e8ed6004d068b6b7a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4863221
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
python3 is the only supported version of python in depot_tools.
Changes include:
* updating python version requirements
* removing third_party/six
Bug: 1475402
Change-Id: Ib86231413fcac494ceb27d1783e53b9d86533c26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4845292
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
If git submodules and DEPS ever go out of sync, there are two different
behaviors in gclient:
* gclient gitmodules needs to read DEPS file and generate submodules
* all other gclient operations prefer content of git submodules.
Since we enforce SYNC to always match their content via presubmit
checks, it's reasonable to expect they will only diverge on development
environment. User is more likely to update DEPS file by hand than to
gitlink.
If user updates a gitlink accidentally, gclient sync won't checkout to
content of DEPS which then will require user to take additional steps to
recover.
This change uses DEPS to be source of truth for git dependencies if
git_dependencies == SYNC.
R=aravindvasudev@google.com, jojwang@google.com
Bug: 1476108
Change-Id: I5b22095b5c9ec764ea6bf07310a0e5bba4a71aad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4820459
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Removing some deadcode from depot_tools scripts. More follow-up CLs will
be made after this to remove more dead code.
R=sokcevic,gavinmak
Change-Id: Iad20e9e1737e49c1370248d32c6317cf2be63de4
Bug: 1475776
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4811216
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Use universal newlines when writing to gitmodules. Also, split
update-index command so we don't hit Windows limit for process lenght.
R=jojwang
Bug: 1475328
Change-Id: Ia9bba4620a276c31081a9773bac8f6bd309bb86e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4808507
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Joanna Wang
Josip Sokcevic
Stephanie Kim
Michael Ershov
Gaby Baghdadi
Aravind Vasudevan
Junji Watanabe
Alexei Svitkine
Gavin Mak
Takuto Ikuta
Philipp Wollermann
Philipp Wollermann
Yiwei Zhang
Mike Frysinger