This is a reland of commit 7d95eb2eb0
Additional changes:
* introduce deletion blocklist
Original change's description:
> [gclient] Delete GCS output_dir on download
>
> GCS output_directory needs to be cleaned up prior to extracting new
> content. At the same time, it can't be cleaned up by individual GCS
> fetcher as there can be multiple objects in the same output_dir.
>
> This change adds a top level support to delete old GCS output_dir. If
> any of objects need to be downloaded, the output_dir will be completely
> deleted and all objects will be redownloaded.
>
> R=jojwang@google.com
>
> Bug: 342522902, 338612245
> Change-Id: Icbe4f1238cac54d7390bbb9b6fc5f17c538cca62
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5570466
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
R=jojwang@google.com
Bug: 342522902, 338612245
Change-Id: Ib5335cddfd60fb4d7da54e16aacb71b11413108e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5581228
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 7d95eb2eb0.
Reason for revert: Causing bot_update failures: https://issues.chromium.org/issues/342945505
Original change's description:
> [gclient] Delete GCS output_dir on download
>
> GCS output_directory needs to be cleaned up prior to extracting new
> content. At the same time, it can't be cleaned up by individual GCS
> fetcher as there can be multiple objects in the same output_dir.
>
> This change adds a top level support to delete old GCS output_dir. If
> any of objects need to be downloaded, the output_dir will be completely
> deleted and all objects will be redownloaded.
>
> R=jojwang@google.com
>
> Bug: 342522902, 338612245
> Change-Id: Icbe4f1238cac54d7390bbb9b6fc5f17c538cca62
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5570466
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bug: 342522902, 338612245, 342945505
Change-Id: I93ed99d7311f15a7a24a03ccba8c310cc7e4c4d4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5570280
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: danakj <danakj@chromium.org>
Reviewed-by: Yann Dago <ydago@google.com>
Reviewed-by: Friedrich Horschig <fhorschig@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
Owners-Override: danakj <danakj@chromium.org>
GCS output_directory needs to be cleaned up prior to extracting new
content. At the same time, it can't be cleaned up by individual GCS
fetcher as there can be multiple objects in the same output_dir.
This change adds a top level support to delete old GCS output_dir. If
any of objects need to be downloaded, the output_dir will be completely
deleted and all objects will be redownloaded.
R=jojwang@google.com
Bug: 342522902, 338612245
Change-Id: Icbe4f1238cac54d7390bbb9b6fc5f17c538cca62
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5570466
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
We now ask users to preserve the order of objects since there are now
two fields that don't need to updated (output_file and condition).
R=kimstephanie@google.com
Change-Id: Ic2e8c0b350060247458e733f14466145c2af435f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5539331
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
We need to preserve the words before the slashes for instances like
`Linux_x64/rust-toolchain-31e6e8c6c5b6ce62656c922c7384d3376018c980-2-llvmorg-19-init-9433-g76ea5feb.tar.xz`
and `Mac/rust-toolchain-31e6e8c6c5b6ce62656c922c7384d3376018c980-2-llvmorg-19-init-9433-g76ea5feb.tar.xz`
Bug: 328065301
Change-Id: I0d745ee8be7751149c292f2bd2f79df2af593bdd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5536288
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
reason: git submodule status command could be slower and output may
not be deterministic depending on the init status and current status
Change-Id: I3c9b8128e6fb230d51eb456234396a557b61c86f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5528758
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
llvm-build GCS dependency needs to place one GCS object based on
platform. We can't have multiple top-level GCS llvm-build dependencies
as the pathname is used as a key.
This extends object property of GCS dependency to have `condition`.
The `condition` field will be AND with the top level condition.
R=jojwang@google.com, kimstephanie@google.com
Change-Id: I06a6281140a8057d598a6eac2b07c7577f0ea86c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5527474
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
So that it can be shared with code other than scm.
Bug: 339231299
Change-Id: Iba68cc7ebba78c8933598d073861290bb0157c40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5529883
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 58ff1badf9.
This change also restricts diff only to changes that are submodule related by
utilizing patch diff search.
Original change's description:
> [gclient] Read submodule status information
>
> This allow us to skip sync if we know the state is correct.
>
> R=gavinmak@google.com
>
> Bug: 40283612, 40942309
> Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 40283612, 40942309
Change-Id: Iac7ed8c927de1a03a3d60dd50108ddb0b6d96c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5513190
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
If a child repository is not cloned, gclient will assume git_dependency
to match parent, which can be incorrect.
R=gavinmak
Change-Id: Icb464524f5dcf2e18ccae1db00c69963ffd9ef02
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506655
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
The number of revision objects must match the number of current objects
for a given path. The comma separated values for a revision object
must also be in the order of object_name, sha256sum, size_bytes, and
generation.
Example usage:
gclient setdep --revision=src/third_party/js_code_coverage@myobjectname,deadbeef,1223455,9438393984/myobjectname2,deadbefeaef,1223455,9438393984
Resulting diff:
diff --git a/DEPS b/DEPS
index b40bca8cdd89d..c2186185c1db5 100644
--- a/DEPS
+++ b/DEPS
@@ -553,10 +553,10 @@ deps = {
'bucket': 'chromium-nodejs',
'objects': [
{
- 'object_name': 'js_code_coverage/d538975c93eefc7bafd599b50f867e90c1ef17f3',
- 'sha256sum': '646bb00ced0a930b2eb1e4dbcfac18ebbb8f889bb80599e0254d9d6505427914',
- 'size_bytes': 1469185,
- 'generation': 1657780123604338,
+ 'object_name': 'myobjectname',
+ 'sha256sum': 'deadbeef',
+ 'size_bytes': 1223455,
+ 'generation': 9438393984,
},
],
},
Bug: b/324418194
Change-Id: Ibd824f7b51fa88f732c7197e2cc663b58de7479e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5503101
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Most tarfiles have .getnames() return top level dirs as <top_level_dir_name>
and files as <top_level_dir_name>/filename.txt. But some tarfiles don't
return the top level directories when running .getnames(). Instead we'll
have to parse through each path name to retrieve the top level directories.
Bug: b/337331070, b/337330736
Change-Id: I8c0e05a2c0722c02eda55162bf51b15a9c21e34a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5499051
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Sometimes the object name can have more than one dot like
20.11.0/f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.
Currently, the dotfiles would be .20_.*
This change makes it so that the dotfiles would be
.20_11_0_f9a337cfa0e2b92d3e5c671c26b454bd8e99769e.*
Bug: b/328065301
Change-Id: I043a4b5fd56e48b046cfdc9a2fde3204b1bb638c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5491861
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The downloaded file may be a binary that needs to be executed
Bug: b/328065301, b/336843583
Change-Id: I0f6c959cbe3d5cf7e2717da72c98c51bbe98f7b3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5484340
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
A user was seeing an error about the output_file not existing. I was
not able to reproduce it on my workstation or sshrimp's.
Raise error if output file does not exist after downloading.
Bug: b/336509491, b/324418194
Change-Id: I4b1c2049d8369304cd2accd677f6fe2e59bcb513
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5483576
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This is so that the third_party directories can gitignore the
downloaded tarfile. Currently theyre ignoring on *tar.gz
Bug: b/324418194
Change-Id: Id4aeeec366de3a0cec410de6ddd63369a3a4680a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454165
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
The sysroot tarfile's getnames() and getmembers() functions returns
paths that begin with './' and a '.'. Removing the
leading './' helps determine which of them are probable top level
directories. Also ignore the '.'
Bug: b/324418194
Change-Id: I30667b685ee85e55f0146a743701aafb4094c928
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5437002
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
When installing a gcs dep entry, record the getnames() output to
.<object_name>_content_names. When updating the object(s) of a gcs
entry, it'll clear all files related to that gcs entry including:
.<object_name>_content_names and the paths inside it,
.<object_name>_hash, and .<object_name>_is_first_class_gcs
Bug: b/324418194
Change-Id: Ieb56623ce929b715ed103af9560efcf66b46a9c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454646
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Now that multiple objects can share a directory, when objects are
removed, the directory should also remove the extracted contents
from that specific object. Since those exact contents are unknown, the
whole directory will be cleared.
If an entire GCS dep is added or removed, the corresponding directory
path will be cleared as well.
.gcs_entries holds a record of which GCS deps and objects
have been downloaded, per checkout. Example:
```
{
"src": {
"src/third_party/llvm-build/Release+Asserts": [
"Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz",
"Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz"
],
"src/third_party/node/linux": [
"46795170ff5df9831955f163f6966abde581c8af"
]
}
}
```
Bug: b/324418194
Change-Id: Icac113572523b61c83450880615418bf7df8bba7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5407888
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This script allows users to upload one file or one or more compressed
directories to GS. It will also output the DEPS blob with the required
sha256sum and size_bytes.
Verified script locally.
Bug: b/328065301
Change-Id: Ic8e894917eb4efde4f287ffb6869a22be1e89bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381863
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This utilizes the same tarfile validation as
download_from_google_storage, but checks that the links and contents
are inside the tarfile's multiple top level directories instead of
just one directory inside the tarfile.
Bug: b/324418194
Change-Id: I73505c72084d54ca8088213dbc40b371675f8dc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381117
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
GCS deps will require a size_bytes field of type `int`. If gclient
sees that the actual byte size is different than the noted size, it
will raise an exception.
Bug: b/328065441
Change-Id: If61496eae39cd372966d7e64ba4a84759708c60b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5367540
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Joanna Wang
Josip Sokcevic
danakj
Yiwei Zhang
Stephanie Kim
Michael Ershov