Commit Graph

500 Commits (56edc1d23df5c72021b74dcabe6b8b05abe5643a)

Author SHA1 Message Date
Josip Sokcevic 4c9d29b23b Reland "[gclient] Read submodule status information
This reverts commit 58ff1badf9.

This change also restricts diff only to changes that are submodule related by
utilizing patch diff search.

Original change's description:
> [gclient] Read submodule status information
>
> This allow us to skip sync if we know the state is correct.
>
> R=gavinmak@google.com
>
> Bug: 40283612, 40942309
> Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>

Bug: 40283612, 40942309
Change-Id: Iac7ed8c927de1a03a3d60dd50108ddb0b6d96c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5513190
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
12 months ago
Josip Sokcevic 58ff1badf9 Revert "Reland "[gclient] Read submodule status information""
This reverts commit 452cb7b6cb.

Reason for revert: v8 deps roll failed, gclient revinfo is
misbehaving.

Original change's description:
> Reland "[gclient] Read submodule status information"
>
> This reverts commit c0366630f7.
>
> Original change's description:
> > Revert "[gclient] Read submodule status information"
> >
> > This reverts commit 39501ba8c9.
> >
> > Reason for revert: broke submodule update builder: gclient revinfo
> >
> >
> >
> > Original change's description:
> > > [gclient] Read submodule status information
> > >
> > > This allow us to skip sync if we know the state is correct.
> > >
> > > R=gavinmak@google.com
> > >
> > > Bug: 40283612, 40942309
> > > Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> > > Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> > > Reviewed-by: Gavin Mak <gavinmak@google.com>
> >
> > Bug: 40283612, 40942309
> > Change-Id: Iaf478c65eb6f18b19afc3fdda2996190bdc58613
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506653
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
> > Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
>
> Bug: 40283612, 40942309
> Change-Id: I707b0eb1d3850c415e474df8cd43f561054017cb
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506833
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>

Bug: 40283612, 40942309
Change-Id: Id667a0e71e6e0a0e74c98ea96952a96c024e3289
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5513187
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
12 months ago
Yiwei Zhang 858263702e add GetsubmoduleDiff and GetSubmoduleStateFromIndex to CogWrapper
Without it, runnign gclient in Cog would fail with error
"AttributeError: 'CogWrapper' object has no attribute 'GetSubmoduleDiff'"

Change-Id: I6a3220f882eb38da9ebc59b6cac67743c41f4c7a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5509813
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
12 months ago
Josip Sokcevic 452cb7b6cb Reland "[gclient] Read submodule status information"
This reverts commit c0366630f7.

Original change's description:
> Revert "[gclient] Read submodule status information"
>
> This reverts commit 39501ba8c9.
>
> Reason for revert: broke submodule update builder: gclient revinfo
>
>
>
> Original change's description:
> > [gclient] Read submodule status information
> >
> > This allow us to skip sync if we know the state is correct.
> >
> > R=gavinmak@google.com
> >
> > Bug: 40283612, 40942309
> > Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> > Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
>
> Bug: 40283612, 40942309
> Change-Id: Iaf478c65eb6f18b19afc3fdda2996190bdc58613
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506653
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Bug: 40283612, 40942309
Change-Id: I707b0eb1d3850c415e474df8cd43f561054017cb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506833
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
12 months ago
Josip Sokcevic c0366630f7 Revert "[gclient] Read submodule status information"
This reverts commit 39501ba8c9.

Reason for revert: broke submodule update builder: gclient revinfo



Original change's description:
> [gclient] Read submodule status information
>
> This allow us to skip sync if we know the state is correct.
>
> R=gavinmak@google.com
>
> Bug: 40283612, 40942309
> Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>

Bug: 40283612, 40942309
Change-Id: Iaf478c65eb6f18b19afc3fdda2996190bdc58613
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5506653
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
12 months ago
Josip Sokcevic 39501ba8c9 [gclient] Read submodule status information
This allow us to skip sync if we know the state is correct.

R=gavinmak@google.com

Bug: 40283612, 40942309
Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
12 months ago
Josip Sokcevic 5a0631fd9d [gclient] Reduce number of git operations during sync
This version is 1.08 ± 0.04 times faster based on hyperfine results.

Bug: 40942309
Change-Id: If67e53bb778b88b0cb7c44ad86ba7438ab843ce3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480171
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
1 year ago
Stephanie Kim 336643a14b Don't clear directory for new/removed entries
This was from back when we were clearing the whole directory instead
of just the extracted contents

Bug: b/324418194
Change-Id: I496b0faed2023a8fa3b4ea772a7744f2acc9be6a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5459298
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
1 year ago
Stephanie Kim 8b40b1b381 Delete paths inside .<object-name>_content_names
When installing a gcs dep entry, record the getnames() output to
.<object_name>_content_names. When updating the object(s) of a gcs
entry, it'll clear all files related to that gcs entry including:
.<object_name>_content_names and the paths inside it,
.<object_name>_hash, and .<object_name>_is_first_class_gcs

Bug: b/324418194
Change-Id: Ieb56623ce929b715ed103af9560efcf66b46a9c4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454646
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Stephanie Kim 589ccd8d54 Add GCS root and .gcs_entries file to keep track of installed gcs deps
Now that multiple objects can share a directory, when objects are
removed, the directory should also remove the extracted contents
from that specific object. Since those exact contents are unknown, the
whole directory will be cleared.
If an entire GCS dep is added or removed, the corresponding directory
path will be cleared as well.

.gcs_entries holds a record of which GCS deps and objects
have been downloaded, per checkout. Example:
```
{
  "src": {
    "src/third_party/llvm-build/Release+Asserts": [
      "Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz",
      "Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz"
    ],
    "src/third_party/node/linux": [
      "46795170ff5df9831955f163f6966abde581c8af"
    ]
  }
}
```

Bug: b/324418194
Change-Id: Icac113572523b61c83450880615418bf7df8bba7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5407888
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
1 year ago
Stephanie Kim 63ae5028c1 Reland "Reland "Add support for GCS deps""
This is a reland of commit 46d5382f69

Whats fixed: Removed the call_google_storage refactor so
that all `import download_from_google_storage` callsites
(in depot_tools and chromium/src) won't be affected.
Instead, gclient.py will import Gsutil from download_from_google_storage.

Original change's description:
> Reland "Add support for GCS deps"
>
> This is a reland of commit 3eedee7b55
>
> Fixed more imports in download_google_storage.py
>
> Original change's description:
> > Add support for GCS deps
> >
> > Also take out GCS calling logic from download_google_storage and
> > into call_google_storage.
> >
> > GCS deps look like:
> >    'src/third_party/node/linux': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-nodejs/20.11.0',
> >        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
> >        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
> >    },
> >
> >    'src/third_party/llvm-build/Release+Asserts': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-browser-clang',
> >        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
> >        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
> >    },
> >
> > Example directory for src/third_party/node/linux after gclient sync:
> > - tar_file.gz is the downloaded file from GCS.
> > - node_linux_x64/ is extracted in its path.
> > - `hash` contains the sha of GCS filename.
> > ```
> > chromium/src/ ->
> >    third_party/node/linux/ ->
> >        hash, tar_file.gz, node_linux_x64/
> > ```
> >
> > Bug: b/324418194
> > Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> > Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
>
> Bug: b/324418194
> Change-Id: Ie64265a86abcec0135408715a45c32a8bb7c7408
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5346338
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>

Bug: b/324418194
Change-Id: I8b58dadbaa740fd9da1fbaf29b3b6ff5ef67fd12
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5352896
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Michael Ershov 3c79cfc485 Revert "Reland "Add support for GCS deps""
This reverts commit 46d5382f69.

Reason for revert: win-presubmit builder probably started failing after this CL.
https://ci.chromium.org/ui/p/chromium/builders/ci/win-presubmit

Original change's description:
> Reland "Add support for GCS deps"
>
> This is a reland of commit 3eedee7b55
>
> Fixed more imports in download_google_storage.py
>
> Original change's description:
> > Add support for GCS deps
> >
> > Also take out GCS calling logic from download_google_storage and
> > into call_google_storage.
> >
> > GCS deps look like:
> >    'src/third_party/node/linux': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-nodejs/20.11.0',
> >        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
> >        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
> >    },
> >
> >    'src/third_party/llvm-build/Release+Asserts': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-browser-clang',
> >        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
> >        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
> >    },
> >
> > Example directory for src/third_party/node/linux after gclient sync:
> > - tar_file.gz is the downloaded file from GCS.
> > - node_linux_x64/ is extracted in its path.
> > - `hash` contains the sha of GCS filename.
> > ```
> > chromium/src/ ->
> >    third_party/node/linux/ ->
> >        hash, tar_file.gz, node_linux_x64/
> > ```
> >
> > Bug: b/324418194
> > Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> > Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
>
> Bug: b/324418194
> Change-Id: Ie64265a86abcec0135408715a45c32a8bb7c7408
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5346338
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>

Bug: b/324418194
Change-Id: Ic4517f6c9e05aea3f3f052d2a44865733236998b
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5352167
Auto-Submit: Michael Ershov <miersh@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Owners-Override: Michael Ershov <miersh@google.com>
1 year ago
Stephanie Kim 46d5382f69 Reland "Add support for GCS deps"
This is a reland of commit 3eedee7b55

Fixed more imports in download_google_storage.py

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: Ie64265a86abcec0135408715a45c32a8bb7c7408
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5346338
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
1 year ago
Stephanie Kim 93d5662c1d Revert "Reland "Add support for GCS deps""
This reverts commit d94a03f21e.

Reason for revert: PLATFORM_MAPPING import issue

Original change's description:
> Reland "Add support for GCS deps"
>
> This is a reland of commit 3eedee7b55
>
> Original change's description:
> > Add support for GCS deps
> >
> > Also take out GCS calling logic from download_google_storage and
> > into call_google_storage.
> >
> > GCS deps look like:
> >    'src/third_party/node/linux': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-nodejs/20.11.0',
> >        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
> >        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
> >    },
> >
> >    'src/third_party/llvm-build/Release+Asserts': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-browser-clang',
> >        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
> >        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
> >    },
> >
> > Example directory for src/third_party/node/linux after gclient sync:
> > - tar_file.gz is the downloaded file from GCS.
> > - node_linux_x64/ is extracted in its path.
> > - `hash` contains the sha of GCS filename.
> > ```
> > chromium/src/ ->
> >    third_party/node/linux/ ->
> >        hash, tar_file.gz, node_linux_x64/
> > ```
> >
> > Bug: b/324418194
> > Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> > Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
>
> Bug: b/324418194
> Change-Id: I8b960f6dd3534658a9ee0b7688e68eaa15df1444
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5334791
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: I28bc0ba838a1135ff882cd88b77dbb2cb4b85b82
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5345022
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
1 year ago
Stephanie Kim d94a03f21e Reland "Add support for GCS deps"
This is a reland of commit 3eedee7b55

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: I8b960f6dd3534658a9ee0b7688e68eaa15df1444
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5334791
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Stephanie Kim 064e03aa1c Revert "Add support for GCS deps"
This reverts commit 3eedee7b55.

Reason for revert: https://buganizer.corp.google.com/issues/324418194#comment7

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: Ie74df90065c1fea087b240d98005b9d9b4f44411
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5336079
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Stephanie Kim 3eedee7b55 Add support for GCS deps
Also take out GCS calling logic from download_google_storage and
into call_google_storage.

GCS deps look like:
   'src/third_party/node/linux': {
       'dep_type': 'gcs',
       'condition': 'checkout_linux',
       'bucket': 'chromium-nodejs/20.11.0',
       'object_name': '46795170ff5df9831955f163f6966abde581c8af',
       'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
   },

   'src/third_party/llvm-build/Release+Asserts': {
       'dep_type': 'gcs',
       'condition': 'checkout_linux',
       'bucket': 'chromium-browser-clang',
       'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
       'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
   },

Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
   third_party/node/linux/ ->
       hash, tar_file.gz, node_linux_x64/
```

Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Aravind Vasudevan 7571749aad [gclient_scm] Use --replace-all within set_config
Bug: 40942309
Change-Id: Iee6473e73fa7a0ce7ec33640c0d2fe8f49054c3a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5292943
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Aravind Vasudevan 9f9bab25da Reland "Update gclient to use git config caching"
This reverts commit 3569608028.

Reason for revert: This includes a fix for crbug.com/324358728.

The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.

[1] https://codereview.chromium.org/228353003

Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Aravind Vasudevan 3569608028 Revert "Update gclient to use git config caching"
This reverts commit 3edda8d185.

Reason for revert: Breaks rebase-update; crbug.com/324358728

Original change's description:
> Update gclient to use git config caching
>
> This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
>
> This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
>
> A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
>
> Benchmarking
> ============
> With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
>
> Windows Benchmark
> =================
> Baseline (gpaste/6360045736951808): ~1min 12 sec.
> With Caching (gpaste/6480065209040896): ~1min 3sec.
> ~12.5% decrease in gclient sync noop runtime.
>
> Linux Benchmark
> ===============
> Baseline (gpaste/4730436763254784): ~3.739 sec.
> With Caching (gpaste/4849870978940928): ~3.534 sec.
> ~5.5% decrease in gclient sync noop runtime.
>
> Bug: 1501984
> Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Bug: 1501984
Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Aravind Vasudevan 3edda8d185 Update gclient to use git config caching
This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.

This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.

A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.

Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.

Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.

Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.

Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Nikita Leshenko dbbe224787 gclient: Call _AutoFetchRef before using a rev in _FetchAndReset
Other places in gclient_scm (GitWrapper._Clone, GitWrapper.update) do
the same.

Note that we call _AutoFetchRef even after running _Fetch, because
_Fetch in not guaranteed to fetch a specific revision. For example, a
checkout might be configured without branch heads and without tags, yet
a revision might point to a tag.

Change-Id: I5cfd8ebafa4b45afd72b0251f04e5d713199f2fd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4715239
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
1 year ago
Josip Sokcevic 751c658593 [gclient] Reuse rev-parse output in verbose mode
If verbose mode is used, we call rev-parse x2 for each git dependency.
Reusing the value results in 10% reduction for gclient no-op sync (~2s
savings).

R=gavinmak@google.com

Change-Id: I7d1a0a13f2aea3df20b25816fef22ef777a12bf5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5146995
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Aravind Vasudevan 259774c3ac [gclient_scm] Update _CheckClean() to use `git status`
This change updates the `_CheckClean()` fn to use `git status` instead
of `git update-index` and `git diff-index`. The `_CheckClean()` fn is
run during every update and this change reduces the subprocess calls
made by it by half.

Change-Id: Ie9a23b0bce748bec4cac88df09655569e88e4841
Bug: 1501984
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5076224
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
1 year ago
Josip Sokcevic 4cf6518959 scm: Remove usage of distutil
The package is deprecated and will be removed in py 3.12.

R=ddoman@google.com

Bug: 1500415
Change-Id: Ia130c556ad3f2135fac4a9b1d9bb4ecc088cfe9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5030818
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jonathan Nieder <jrn@google.com>
1 year ago
Josip Sokcevic 8fb358e5b6 Add support for Cog SCM
Change-Id: I450e1ce7943fda55edf1cb8016f201ce08cd9251
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5018522
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
1 year ago
Josip Sokcevic 88cc0b8ca8 Remove old Cog detection
This is no longer needed.

R=aravindvasudev@google.com

Change-Id: Ic1b2bbf0f00ca9d288c21fa1623c4ada80d59644
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5014581
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Philipp Thiel f2449cd841 Fix accessing of non-existent exception attribute in error handler
Fixed: 1488385
Change-Id: Ie401b1580091aa1ad31091e71e55b59e26e6fdc6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4931232
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
2 years ago
Joanna Wang 4af2818c5b Fix path's current and new url comparison.
Steps to reproduce:
https://bugs.chromium.org/p/chromium/issues/detail?id=1484613#c2

Bug: 1484613
Change-Id: Id42e8e8019334444d6b2423550f04f3869e43bbd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4875677
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
2 years ago
Josip Sokcevic 3b9212b7ee [gclient_scm] Disable push recurseSubmodule
This option can lead to high `git cl upload` latency. We never want to
use this option for gclient managed repositories anyways since changes
need to go through code review.

Note: the default value is off, but it will be set to on if
`submodule.recurse` is set to true.

R=jojwang@google.com

Bug: b/299969165
Change-Id: I2173a2d96d478bf96bfe6e78ef0c5b21d4ac8015
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4874177
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Yiwei Zhang 523537049c gclient: choose cipd log-level based on the gclient verbosity
R=sokcevic

Bug: 1483244
Change-Id: Ia344386c5b59790c32160bab2b11a428c5b9290f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4869692
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak 7f5b53ff8d Remove last py2 workarounds
python3 is the only supported version of python in depot_tools.

Changes include:
  * updating python version requirements
  * removing third_party/six

Bug: 1475402
Change-Id: Ib86231413fcac494ceb27d1783e53b9d86533c26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4845292
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Yiwei Zhang 28116103e4 change the log level of cipd ensure command to info
I suspect the reason of INFRA_FAILURE in https://crbug.com/1477933#c10
is that cipd is busy with downloading on the packages (it's also
possible that cipd is stucked on one or more packages) and since the
log-level is error, it won't ouptut anything until it completes.
However, from bot_update PoV, gclient is considered as a stale process
because it won't write anything to stdout in 20min so it decide to kill
the process. This CL fix the issue by changing the log-level to info
instead.

R=sokcevic

Bug: 1477933
Change-Id: I2fa05bcfe7f63debeda3371e755b5e7c6d011f12
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4844365
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 1a8bf9e370 Improve error message on gclient sync failure
The new error message should clarify what we expect from user to do.

Also, git v2.42.0 introduced new advice on merge failure, which breaks
existing regex match. Instead, use search.

R=aravindvasudev@google.com

Fixed: 1475622
Change-Id: Ib4e60947aaa7994715cfae3260dd27f85f2380b4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4823716
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
2 years ago
Aravind Vasudevan ed935cf29b Remove deadcode from depot_tools scripts
Removing some deadcode from depot_tools scripts. More follow-up CLs will
be made after this to remove more dead code.

R=sokcevic,gavinmak

Change-Id: Iad20e9e1737e49c1370248d32c6317cf2be63de4
Bug: 1475776
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4811216
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Gavin Mak 65c49b18b9 Drop py2 support in gclient and related files
python3 is the only supported version of python in depot_tools.

Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Joanna Wang d9ea072747 Use diff.ignoreSubmodules=dirty.
With this, modified files from submodules won't be reported,
but new ones will and this makes things much faster than not
having diff.ignoreSubmodules set at all.

Bug: 1470194
Change-Id: I041b6fb3151c0b06cff3cc77d6735ad2ebe27f21
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4793910
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Joanna Wang <jojwang@chromium.org>
2 years ago
Joanna Wang 4e6c1071bd Unset diff.ignoreSubmodules=all when running gclient sync.
Bug: 1470195
Change-Id: I1317e5f12d6cc7fae44b2a463ddf6d0f85629759
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4767516
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
2 years ago
Josip Sokcevic 84c0bbaf04 Disable git's recurse submodule fetch
Git fetches submodules sequentially by default, which slows down gclient
sync. Instead of relying on git to fetch submodules, we can just
continue to rely on gclient to do the right thing.

R=jojwang@google.com

Bug: 1471702
Change-Id: Id0b104e76f702752bd7fecc7edac9245624bbb90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4766998
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Kenneth Russell 02e70b4357 Handle error conditions when joining scm.GIT.RefToRemoteRef.
Detect failure of scm.GIT.RefToRemoteRef and raise an error before
attempting to join its results as a string. This makes certain
repository errors diagnosable.

Fixed: 1470430
Change-Id: I19cd86d9ed088f5cd511e4c9d7dbe5844165c7cf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4754237
Auto-Submit: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
2 years ago
Joanna Wang a654ff36bb Convert branch revision to remote branch in status call.
We pull the revision from the DEPS url, but if the revision is a branch,  the string must be converted into the local branch names we have for remote branches.

Bug: 1464985
Change-Id: I32c89c947319a04421c46d09d057e42043594e76
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4699021
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
2 years ago
Joanna Wang e1753f63df Set ignoreSubmodule=all for all repos.
Bug: 1415507

Change-Id: I840508c6972e4640793a12aba287ed8dedfe64c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4641611
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
2 years ago
Dan Le Febvre 456d085e75 Adding batch CIPD version resolution to gclient.
At the moment on repo's with many CIPD packages the number
of CLI calls to 'cipd describe' can make it very slow.

This mechanism will call cipd ensure-file-resolve with the entire
list in one call then reference items from the list before trying
individual cipd describe commands.

Bug:b/279097790
Change-Id: Ieb4bc77853357cfadb3ea6311fa0013aa7999ac6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4551125
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Rachael Newitt <renewitt@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
2 years ago
Dan Le Febvre 6316ac234e [ssci] adding gclient testing support for the CIPD expand package cmd
Bug:b/279097790
Change-Id: I1edf1c33333bf4ddd32bf462f5d23733097c25d3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4529697
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
2 years ago
Alexander Thomas 24996afd00 Revert "Reland "resolve CIPD package names in gclient""
This reverts commit d69302761b.

Reason for revert: Broke Dart CI
```
gclient.py revinfo --output-json output.json
   --filter='https://chrome-infra-packages.appspot.com/dart/dart-sdk/${platform}'
```

No longer returns:
```
 {
   "sdk/tools/sdks/dart-sdk:dart/dart-sdk/${platform}": {
     "rev": "git_revision:7a6514d1377175decd3a886fe4190fbbebddac3a", 
     "url": "https://chrome-infra-packages.appspot.com/dart/dart-sdk/${platform}"
   }
 }
```

Bad build:
https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-aot-msan-linux-release-x64/90/overview
Good build:
https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-aot-msan-linux-release-x64/89/overview

Original change's description:
> Reland "resolve CIPD package names in gclient"
>
> This CL relands the below CL with the addition of handling situations where the CIPD package name has a variable=value and does not equate to true. In this case just return the original value in DEPS.
>
> This is a reland of commit 8faa5514ec
>
> Original change's description:
> > resolve CIPD package names in gclient
> >
> > Currently the package names are not resolved in gclient output, see example below.
> >
> > This is part of broader work to improve CIPD output, the next CL will replace 'None'.
> >
> > ```
> > $~ gclient revinfo -a | grep '${platform}\|${arch}'
> >
> > src/buildtools/linux64:gn/gn/linux-${arch}: None
> > src/buildtools/reclient:infra/rbe/client/${platform}: None
> > src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> > src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
> > src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> > src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> > src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> > src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
> > src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
> > src/tools/resultdb:infra/tools/result_adapter/${platform}: None
> > ```
> >
> > Bug:b/279097790
> > Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
> > Reviewed-by: Rachael Newitt <renewitt@google.com>
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Commit-Queue: Dan Le Febvre <dlf@google.com>
>
> Bug: b/279097790
> Change-Id: Ib08dac506ce221243c595dde5cb00e8e0d7dc7ed
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4501582
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Dan Le Febvre <dlf@google.com>

Bug: b/279097790
Change-Id: Ibf82354ac5005d9d6279d88aa99c8fb344aa6833
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4518113
Owners-Override: Daniel Cheng <dcheng@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Dan Le Febvre <dlf@google.com>
2 years ago
Dan Le Febvre d69302761b Reland "resolve CIPD package names in gclient"
This CL relands the below CL with the addition of handling situations where the CIPD package name has a variable=value and does not equate to true. In this case just return the original value in DEPS.

This is a reland of commit 8faa5514ec

Original change's description:
> resolve CIPD package names in gclient
>
> Currently the package names are not resolved in gclient output, see example below.
>
> This is part of broader work to improve CIPD output, the next CL will replace 'None'.
>
> ```
> $~ gclient revinfo -a | grep '${platform}\|${arch}'
>
> src/buildtools/linux64:gn/gn/linux-${arch}: None
> src/buildtools/reclient:infra/rbe/client/${platform}: None
> src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
> src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
> src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
> src/tools/resultdb:infra/tools/result_adapter/${platform}: None
> ```
>
> Bug:b/279097790
> Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
> Reviewed-by: Rachael Newitt <renewitt@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Dan Le Febvre <dlf@google.com>

Bug: b/279097790
Change-Id: Ib08dac506ce221243c595dde5cb00e8e0d7dc7ed
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4501582
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
2 years ago
Dan Le Febvre f2da906faf Revert "resolve CIPD package names in gclient"
This reverts commit 8faa5514ec.

Reason for revert: cipd expand-package-name errors on wrong platform

Original change's description:
> resolve CIPD package names in gclient
>
> Currently the package names are not resolved in gclient output, see example below.
>
> This is part of broader work to improve CIPD output, the next CL will replace 'None'.
>
> ```
> $~ gclient revinfo -a | grep '${platform}\|${arch}'
>
> src/buildtools/linux64:gn/gn/linux-${arch}: None
> src/buildtools/reclient:infra/rbe/client/${platform}: None
> src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
> src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
> src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
> src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
> src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
> src/tools/resultdb:infra/tools/result_adapter/${platform}: None
> ```
>
> Bug:b/279097790
> Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
> Reviewed-by: Rachael Newitt <renewitt@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Dan Le Febvre <dlf@google.com>

Bug: b/279097790
Change-Id: I0beba8606d8cceeb815629d6eae6d8021c7731a8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4501579
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Auto-Submit: Dan Le Febvre <dlf@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Owners-Override: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
2 years ago
Dan Le Febvre 8faa5514ec resolve CIPD package names in gclient
Currently the package names are not resolved in gclient output, see example below.

This is part of broader work to improve CIPD output, the next CL will replace 'None'.

```
$~ gclient revinfo -a | grep '${platform}\|${arch}'

src/buildtools/linux64:gn/gn/linux-${arch}: None
src/buildtools/reclient:infra/rbe/client/${platform}: None
src/third_party/dawn/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
src/third_party/dawn/tools/golang:infra/3pp/tools/go/${platform}: None
src/third_party/devtools-frontend-internal/devtools-frontend/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
src/third_party/devtools-frontend/src/third_party/esbuild:infra/3pp/tools/esbuild/${platform}: None
src/third_party/ninja:infra/3pp/tools/ninja/${platform}: None
src/tools/luci-go:infra/tools/luci/isolate/${platform}: None
src/tools/luci-go:infra/tools/luci/swarming/${platform}: None
src/tools/resultdb:infra/tools/result_adapter/${platform}: None
```

Bug:b/279097790
Change-Id: I21abf2579910e9d79d9ee0dcd018ee761e3d3c1c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4463983
Reviewed-by: Rachael Newitt <renewitt@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dan Le Febvre <dlf@google.com>
2 years ago