llvm-build GCS dependency needs to place one GCS object based on
platform. We can't have multiple top-level GCS llvm-build dependencies
as the pathname is used as a key.
This extends object property of GCS dependency to have `condition`.
The `condition` field will be AND with the top level condition.
R=jojwang@google.com, kimstephanie@google.com
Change-Id: I06a6281140a8057d598a6eac2b07c7577f0ea86c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5527474
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
So that it can be shared with code other than scm.
Bug: 339231299
Change-Id: Iba68cc7ebba78c8933598d073861290bb0157c40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5529883
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 58ff1badf9.
This change also restricts diff only to changes that are submodule related by
utilizing patch diff search.
Original change's description:
> [gclient] Read submodule status information
>
> This allow us to skip sync if we know the state is correct.
>
> R=gavinmak@google.com
>
> Bug: 40283612, 40942309
> Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 40283612, 40942309
Change-Id: Iac7ed8c927de1a03a3d60dd50108ddb0b6d96c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5513190
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
If a child repository is not cloned, gclient will assume git_dependency
to match parent, which can be incorrect.
R=gavinmak
Change-Id: Icb464524f5dcf2e18ccae1db00c69963ffd9ef02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506655
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
The number of revision objects must match the number of current objects
for a given path. The comma separated values for a revision object
must also be in the order of object_name, sha256sum, size_bytes, and
generation.
Example usage:
gclient setdep --revision=src/third_party/js_code_coverage@myobjectname,deadbeef,1223455,9438393984/myobjectname2,deadbefeaef,1223455,9438393984
Resulting diff:
diff --git a/DEPS b/DEPS
index b40bca8cdd89d..c2186185c1db5 100644
--- a/DEPS
+++ b/DEPS
@@ -553,10 +553,10 @@ deps = {
'bucket': 'chromium-nodejs',
'objects': [
{
- 'object_name': 'js_code_coverage/d538975c93eefc7bafd599b50f867e90c1ef17f3',
- 'sha256sum': '646bb00ced0a930b2eb1e4dbcfac18ebbb8f889bb80599e0254d9d6505427914',
- 'size_bytes': 1469185,
- 'generation': 1657780123604338,
+ 'object_name': 'myobjectname',
+ 'sha256sum': 'deadbeef',
+ 'size_bytes': 1223455,
+ 'generation': 9438393984,
},
],
},
Bug: b/324418194
Change-Id: Ibd824f7b51fa88f732c7197e2cc663b58de7479e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5503101
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Most tarfiles have .getnames() return top level dirs as <top_level_dir_name>
and files as <top_level_dir_name>/filename.txt. But some tarfiles don't
return the top level directories when running .getnames(). Instead we'll
have to parse through each path name to retrieve the top level directories.
Bug: b/337331070, b/337330736
Change-Id: I8c0e05a2c0722c02eda55162bf51b15a9c21e34a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5499051
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Sometimes the object name can have more than one dot like
20.11.0/f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.
Currently, the dotfiles would be .20_.*
This change makes it so that the dotfiles would be
.20_11_0_f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.*
Bug: b/328065301
Change-Id: I043a4b5fd56e48b046cfdc9a2fde3204b1bb638c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5491861
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The downloaded file may be a binary that needs to be executed
Bug: b/328065301, b/336843583
Change-Id: I0f6c959cbe3d5cf7e2717da72c98c51bbe98f7b3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5484340
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
A user was seeing an error about the output_file not existing. I was
not able to reproduce it on my workstation or sshrimp's.
Raise error if output file does not exist after downloading.
Bug: b/336509491, b/324418194
Change-Id: I4b1c2049d8369304cd2accd677f6fe2e59bcb513
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5483576
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This is so that the third_party directories can gitignore the
downloaded tarfile. Currently theyre ignoring on *tar.gz
Bug: b/324418194
Change-Id: Id4aeeec366de3a0cec410de6ddd63369a3a4680a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454165
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The sysroot tarfile's getnames() and getmembers() functions returns
paths that begin with './' and a '.'. Removing the
leading './' helps determine which of them are probable top level
directories. Also ignore the '.'
Bug: b/324418194
Change-Id: I30667b685ee85e55f0146a743701aafb4094c928
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5437002
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
When installing a gcs dep entry, record the getnames() output to
.<object_name>_content_names. When updating the object(s) of a gcs
entry, it'll clear all files related to that gcs entry including:
.<object_name>_content_names and the paths inside it,
.<object_name>_hash, and .<object_name>_is_first_class_gcs
Bug: b/324418194
Change-Id: Ieb56623ce929b715ed103af9560efcf66b46a9c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454646
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Now that multiple objects can share a directory, when objects are
removed, the directory should also remove the extracted contents
from that specific object. Since those exact contents are unknown, the
whole directory will be cleared.
If an entire GCS dep is added or removed, the corresponding directory
path will be cleared as well.
.gcs_entries holds a record of which GCS deps and objects
have been downloaded, per checkout. Example:
```
{
"src": {
"src/third_party/llvm-build/Release+Asserts": [
"Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz",
"Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz"
],
"src/third_party/node/linux": [
"46795170ff5df9831955f163f6966abde581c8af"
]
}
}
```
Bug: b/324418194
Change-Id: Icac113572523b61c83450880615418bf7df8bba7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5407888
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This script allows users to upload one file or one or more compressed
directories to GS. It will also output the DEPS blob with the required
sha256sum and size_bytes.
Verified script locally.
Bug: b/328065301
Change-Id: Ic8e894917eb4efde4f287ffb6869a22be1e89bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381863
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This utilizes the same tarfile validation as
download_from_google_storage, but checks that the links and contents
are inside the tarfile's multiple top level directories instead of
just one directory inside the tarfile.
Bug: b/324418194
Change-Id: I73505c72084d54ca8088213dbc40b371675f8dc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381117
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
GCS deps will require a size_bytes field of type `int`. If gclient
sees that the actual byte size is different than the noted size, it
will raise an exception.
Bug: b/328065441
Change-Id: If61496eae39cd372966d7e64ba4a84759708c60b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5367540
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
No script can run on py2 (due to print syntax change), so those
checks are useless (will never be printed).
Change-Id: I9cc7bc8b2665dd09cc7f8939ba255956f96af88a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5262958
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts commit 92b780b0bd.
Reason for revert: mac arm experiencing hang
Original change's description:
> [gclient] Enable parallel sync on ARM Mac by default
>
> I realized that gclient sync doesn't use all the cores on ARM Mac.
> This CL changes the default for ARM Mac to use all the available cores.
>
> Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
> Commit-Queue: Junji Watanabe <jwata@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Junji Watanabe <jwata@google.com>
Bug: 1427050
Change-Id: I512d1d1ef0034f12fc9054c5234ae688ec281857
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5259766
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
I realized that gclient sync doesn't use all the cores on ARM Mac.
This CL changes the default for ARM Mac to use all the available cores.
Change-Id: Id0316e48d2b16a05a397a69bad53172b3d742f4a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5256559
Commit-Queue: Junji Watanabe <jwata@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Junji Watanabe <jwata@google.com>
Change-Id: Ic2d3ea0dbaad27aad13dcfc6cc3c4f5c1fd8c545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5055942
Reviewed-by: Scott Lee <ddoman@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit 8672d51b79.
Reason for revert: crrev.com/c/5018522 has landed, so this is no longer required.
Original change's description:
> gclient: Add skip-git experiment.
>
> This experiment causes all dependencies of type 'git' to be skipped for
> all commands. This can be useful if you need to build Chromium in a
> working tree that is guaranteed to contain a consistent, pre-synced
> snapshot of the code that's managed by Git, but still need to install
> cipd dependencies and run the hooks to download the remaining
> dependencies.
>
> Bug: b/310485239
> Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Auto-Submit: Philipp Wollermann <philwo@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bug: b/310485239
Change-Id: If1fc93d493daa9515ddfb568aecf4374deb6ec03
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035260
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This experiment causes all dependencies of type 'git' to be skipped for
all commands. This can be useful if you need to build Chromium in a
working tree that is guaranteed to contain a consistent, pre-synced
snapshot of the code that's managed by Git, but still need to install
cipd dependencies and run the hooks to download the remaining
dependencies.
Bug: b/310485239
Change-Id: Iee1e6a4d62b867b6d7235903817f927a9e806dc1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018571
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Yiwei Zhang
Josip Sokcevic
Stephanie Kim
Joanna Wang
Michael Ershov
Gaby Baghdadi
Aravind Vasudevan
Junji Watanabe
Alexei Svitkine
Gavin Mak
Takuto Ikuta
Philipp Wollermann
Philipp Wollermann