Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
GCS deps will require a size_bytes field of type `int`. If gclient
sees that the actual byte size is different than the noted size, it
will raise an exception.
Bug: b/328065441
Change-Id: If61496eae39cd372966d7e64ba4a84759708c60b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5367540
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
.git suffix is not well supported by the systems that utilize
submodules. Since some tools expect DEPS to have .git suffix, it's not
possible to simply update DEPS without breaking the tools.
This patch updates `gclient gitmodules` to omit .git suffix and trailing
slashes. However, if `.gitmodule` already exists, we are expecting the
tool to generate the same result if DEPS didn't change.
R=aravindvasudev@google.com
Bug: b/303296048
Change-Id: I30497b9d5f4e9681e16b4a913151de0f4a7c9545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5008942
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py
Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reformat this dir by itself to help merging with conflicts with other CLs.
Reformatted using:
parallel ./yapf -i -- testing_support/*.py
~/chromiumos/chromite/contrib/reflow_overlong_comments testing_support/*.py
The files that still had strings that were too long were manually
reformatted.
testing_support/coverage_utils.py
testing_support/fake_repos.py
testing_support/git_test_utils.py
testing_support/presubmit_canned_checks_test_mocks.py
Change-Id: I4726a4bbd279a70bcf65d0987fcff0ff9a231386
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4842593
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Off by one - we should have 4 parts, so we need maxsplit=3.
R=aravindvasudev@google.com, jojwang
Fixed: 1474283
Change-Id: I13b568feb8ee2aa307a2267c43765c1749b77c90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4799773
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
At the moment on repo's with many CIPD packages the number
of CLI calls to 'cipd describe' can make it very slow.
This mechanism will call cipd ensure-file-resolve with the entire
list in one call then reference items from the list before trying
individual cipd describe commands.
Bug:b/279097790
Change-Id: Ieb4bc77853357cfadb3ea6311fa0013aa7999ac6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4551125
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Rachael Newitt <renewitt@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
Note: we are only evaluating cipd variables in the URL not the package string on the left hand side, as to not break any existing expected behaviour.
Bug:b/279097790
Change-Id: I0b6008254bd387584b9c0ee083c79bf5dc77fbc8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4536475
Commit-Queue: Dan Le Febvre <dlf@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This reverts commit d69302761b.
Reason for revert: Broke Dart CI
```
gclient.py revinfo --output-json output.json
--filter='https://chrome-infra-packages.appspot.com/dart/dart-sdk/${platform}'
```
No longer returns:
```
{
"sdk/tools/sdks/dart-sdk:dart/dart-sdk/${platform}": {
"rev": "git_revision:7a6514d1377175decd3a886fe4190fbbebddac3a",
"url": "https://chrome-infra-packages.appspot.com/dart/dart-sdk/${platform}"
}
}
```
Bad build:
https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-aot-msan-linux-release-x64/90/overview
Good build:
https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-aot-msan-linux-release-x64/89/overview
Original change's description:
> Reland "resolve CIPD package names in gclient"
>
> This CL relands the below CL with the addition of handling situations where the CIPD package name has a variable=value and does not equate to true. In this case just return the original value in DEPS.
>
> This is a reland of commit 8faa5514ec
>
> Original change's description:
> > resolve CIPD package names in gclient
> >
> > Currently the package names are not resolved in gclient output, see example below.
> >
> > This is part of broader work to improve CIPD output, the next CL will replace 'None'.
> >
> > ```
> > $~ gclient revinfo -a | grep '${platform}\|${arch}'
> >
> > src/buildtools/linux64:gn/gn/linux-${arch}: None
> > src/buildtools/reclient:infra/rbe/client/${platform}: None
> > src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> > src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
> > src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> > src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> > src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> > src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
> > src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
> > src/tools/resultdb:infra/tools/result_adapter/${platform}: None
> > ```
> >
> > Bug:b/279097790
> > Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
> > Reviewed-by: Rachael Newitt <renewitt@google.com>
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Commit-Queue: Dan Le Febvre <dlf@google.com>
>
> Bug: b/279097790
> Change-Id: Ib08dac506ce221243c595dde5cb00e8e0d7dc7ed
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4501582
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Dan Le Febvre <dlf@google.com>
Bug: b/279097790
Change-Id: Ibf82354ac5005d9d6279d88aa99c8fb344aa6833
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4518113
Owners-Override: Daniel Cheng <dcheng@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Dan Le Febvre <dlf@google.com>
This CL relands the below CL with the addition of handling situations where the CIPD package name has a variable=value and does not equate to true. In this case just return the original value in DEPS.
This is a reland of commit 8faa5514ec
Original change's description:
> resolve CIPD package names in gclient
>
> Currently the package names are not resolved in gclient output, see example below.
>
> This is part of broader work to improve CIPD output, the next CL will replace 'None'.
>
> ```
> $~ gclient revinfo -a | grep '${platform}\|${arch}'
>
> src/buildtools/linux64:gn/gn/linux-${arch}: None
> src/buildtools/reclient:infra/rbe/client/${platform}: None
> src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
> src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
> src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
> src/tools/resultdb:infra/tools/result_adapter/${platform}: None
> ```
>
> Bug:b/279097790
> Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
> Reviewed-by: Rachael Newitt <renewitt@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Dan Le Febvre <dlf@google.com>
Bug: b/279097790
Change-Id: Ib08dac506ce221243c595dde5cb00e8e0d7dc7ed
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4501582
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Tests should start to rely on fake_repos.DEFAULT_BRANCH so it can be
renamed in the future. This is a quick fix in case
git.init.defaultBranch is set to a custom value.
Bug: 1144918
Change-Id: I1c8cd591bee6b5fe14a5de4683111a89224cb2b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2521655
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
And merge it in use_relative_paths. In all Chromium repos,
use_relative_hooks is True iff:
- use_relative_paths is True, and
- there is at least one hook
It makes sense that you'd want to use relative hooks if you use relative
paths so this CL merges both flags into use_relative_paths.
Bug: chromium:1107325
Change-Id: I0fe40eba1e1c61be26c812c7ca4329efb72c7f90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2306795
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Corentin Wallez <cwallez@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Without this repos that use the use_relative_paths still need to use
their absolute path for gclient_gn_args_file.
Bug: chromium:1102833
Change-Id: I470096625061a28abf495f4d1035121edbcd581f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2279874
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Running gclient_smoketests times out on windows when using git daemon,
so use a local directory as remote instead.
Bug: 1024683
Change-Id: I6ca506d74de463d914317f176eefbe74996298c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1879723
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
Python doesn't guarantee that the file created by tempfile.NamedTemporaryFile can be opened by multiple programs at once.
This is the case on Windows, so we need to close the temporary PID file and only then pass it to Git.
Bug: 962263
Change-Id: Idf4a36b33ce06d7ab06453c20a690622179aef58
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1608560
Auto-Submit: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Raul Tambre <raul@tambre.ee>
This avoids the need to do the decoding everywhere separately.
Also a small fix for filter() no longer returning a list on Py3.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id0f5153011b2ef1b64394359087864cd9434e45e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595685
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
connect(2) says stream sockets may only succeed to connect once. This means
if the call to sock.connect() fails it will continue failing for the
entirety of the wait loop, and we will fail to wait for a port to bind even
when it eventually becomes available.
Make sure we create a new socket every time we are about to try to connect
to it.
Change-Id: I16d7dbea3590c5bf7f7240bdefcc5ec0bcd1edb5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1528291
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This is a reland of 67ef3f67e8
We no longer call cipd unconditionally.
Original change's description:
> gclient: delete unversioned directory before adding cipd dep for the same path
>
> Bug: 882611
> Change-Id: I46e41cc9693b90874b5d6569a12ec638eaac1050
> Reviewed-on: https://chromium-review.googlesource.com/1228655
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Bug: 882611
Change-Id: I683bfc62bd1eebfec0853583f96f3981c2c6bdf2
Reviewed-on: https://chromium-review.googlesource.com/1232891
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Josip Sokcevic
Stephanie Kim
Joanna Wang
Michael Ershov
Gavin Mak
Mike Frysinger
Dan Le Febvre
Alexander Thomas
Joanna Wang
Aravind Vasudevan
Josip Sokcevic
Corentin Wallez
Quinten Yearsley
Edward Lesmes
Edward Lemur
Marc-Antoine Ruel
Raul Tambre
Raphael Kubo da Costa