I believe this is needed for master->main migration and should no
longer needed.
Change-Id: I75ef8d5bfcf76a862c95c227d6bc9616176c1df1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5540888
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
llvm-build GCS dependency needs to place one GCS object based on
platform. We can't have multiple top-level GCS llvm-build dependencies
as the pathname is used as a key.
This extends object property of GCS dependency to have `condition`.
The `condition` field will be AND with the top level condition.
R=jojwang@google.com, kimstephanie@google.com
Change-Id: I06a6281140a8057d598a6eac2b07c7577f0ea86c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5527474
Reviewed-by: Stephanie Kim <kimstephanie@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Currently, git cl lint runs git commands to retrieve a list of
the affected files and runs the lint tool against the files.
This CL updates the command line argument interface such that,
if file paths are given in positional arguments, git cl will
just run the lint tool against the given files w/o executing any
git commands.
The intent of this CL is to make git cl lint runnable out of Git
checkouts.
Bug: 324595694
Change-Id: I0a68e33931f84c2441da53bf880a2d18a5526ae4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454166
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 58ff1badf9.
This change also restricts diff only to changes that are submodule related by
utilizing patch diff search.
Original change's description:
> [gclient] Read submodule status information
>
> This allow us to skip sync if we know the state is correct.
>
> R=gavinmak@google.com
>
> Bug: 40283612, 40942309
> Change-Id: I30fd5bfb9ca8ab0f7dcce567e2a5cb4aebdc7b2f
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5480172
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
Bug: 40283612, 40942309
Change-Id: Iac7ed8c927de1a03a3d60dd50108ddb0b6d96c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5513190
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
If gsutil is not downloaded, and if gsutil (or
download_from_google_storage) is called concurrently with n>2, it's
possible that those processes are doing the same work. There is also a
critical point where gsutil.py can fail with:
Destination path '/depot_tools_path/external_bin/gsutil/gsutil_4.68/d'
already exists error.
To avoid this problem, use FS locking around code that manipulates with
files.
R=jojwang@google.com
Bug: 338040708
Change-Id: Ib83aaa1e09628f878e512d79f2fa5221c2bcfd37
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5502531
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
If symbolic-ref remote HEAD is not available, we skip setting it and
query remote Git using ls-remote. Such information is not stored, and
gclient will need to repeat it on the next invocation.
Instead, we can call set-head on symbolic-ref failure. While that's
slower operation than ls-remote, it saved in internal Git database and
can be reused on next gclient invocation.
We may be okay with hardcoding 'main' as default remote branch today,
but it's possible that some projects still use old default.
R=gavinmak@google.com
Change-Id: Ic4c826b888d96e367039bfc4b9bd2ba0d8b58b52
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5492789
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
The check relies on cl.GetRemoteBranch() and cl.GetRemoteUrl() that
both rely on git. Provide that info with Gerrit info from input_api
if available instead of always calling git.
Bug: 333744051
Change-Id: I915d6451d808c9bf871804d116ea884294ee7c84
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5479889
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Riley Wong <rgw@google.com>
This simplifies the administration of repositories which is currently
done via the Gerrit UI instead of the CLI.
Bug: None
Change-Id: I0af3d77fb611e0c89dc455a5a215129b5c51c5ea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5472843
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Auto-Submit: Alexander Schulze <alexschulze@chromium.org>
This is so that the third_party directories can gitignore the
downloaded tarfile. Currently theyre ignoring on *tar.gz
Bug: b/324418194
Change-Id: Id4aeeec366de3a0cec410de6ddd63369a3a4680a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5454165
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Always displaying the commit message/git log makes it easier to
upload a manual roll with private details. Omit the git log if rolling
from a non-public host. The `--always-log` flag can be used to include
it anyway.
Bug: 332331835
Change-Id: Ic9201d5323fa04a55dcf4a451a9f76c48eebfd5e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5444190
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Allows multiple objects to use the same directory path.
This requires that each object has its own unique hash file and
unique migration file name. All object names must be unique.
Also update download_from_google_storage to check for the unique
migration file name.
Remove ConvertToGcs smoke tests since we're not converting
any git <-> gcs deps.
Example:
```
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'not llvm_force_head_revision',
'bucket': 'chromium-browser-clang',
'objects': [
{
'object_name': 'Linux_x64/clang-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '7b33138d8592199f97d132242d7b3e10f460c5c9655d49a3ad3767218fba7a77',
'size_bytes': 50212876,
},
{
'object_name': 'Linux_x64/llvmobjdump-llvmorg-19-init-2941-ga0b3dbaf-22.tar.xz',
'sha256sum': '14d669650cd212eb0ccb8c34a9e655338dfdee66fe2ecdaa517f6bd607c09a97',
'size_bytes': 5302312,
},
]
},
```
TODO: update .gitignore to search for *_is_first_class and *_hash
Bug: b/324418194
Change-Id: I89d34b06ee24f4c1aa316cd51530ad078e823143
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5420793
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Include a luci-auth check of the account logged in.
This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.
Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This script allows users to upload one file or one or more compressed
directories to GS. It will also output the DEPS blob with the required
sha256sum and size_bytes.
Verified script locally.
Bug: b/328065301
Change-Id: Ic8e894917eb4efde4f287ffb6869a22be1e89bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381863
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Paths containing ':' characters are illegal only on Windows platforms.
So, ssh_url like git@github.com:chromium/chromium.git will generate
runtime exception on Windows. We replace ':' with '__' to avoid
exceptions in the ssh form of git_url case.
R=ddoman@chromium.org
Bug: 329781388
Change-Id: Ia4db6a0927b40b521477e5d90addb4056c2c8ef8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5369296
Auto-Submit: 吴金立 <wujinli.cn@gmail.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
_process_diff assumes "@@" will always be present in a diff, but those
aren't present in a git diff where only the file mode changed.
Bug: b/323243527
Change-Id: Id129bc6c14994affc43a77feab553fa119612b7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384645
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
There may be other tests which aren't correct though, this is just
the one I saw during presubmit.
R=aravindvasudev@google.com
Bug: 330347045
Change-Id: I6499b5103bf15c30f5e8a2fdcd7abeba094a39d7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5394357
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This flag allows a diff to be generated based on an upstream commit
for provided files and for checks to be run with on that diff.
Bug: b/323243527
Change-Id: Ia3f4ec50b15eabb90d391b9bc795fc4b35a35a08
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376402
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Create a script to create a unified git diff of a local workspace
against a remote Gerrit commit. This will be used to create a diff file
for running presubmits.
Bug: b/323243527
Change-Id: Ia0d287624162dbfe5231a5653c9a962a6c85c8e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5374833
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Also take out GCS calling logic from download_google_storage and
into call_google_storage.
GCS deps look like:
'src/third_party/node/linux': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-nodejs/20.11.0',
'object_name': '46795170ff5df9831955f163f6966abde581c8af',
'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
},
'src/third_party/llvm-build/Release+Asserts': {
'dep_type': 'gcs',
'condition': 'checkout_linux',
'bucket': 'chromium-browser-clang',
'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
},
Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
third_party/node/linux/ ->
hash, tar_file.gz, node_linux_x64/
```
Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This reverts commit 6b84fbfb20.
Reason for revert: Jumped the gun before final resolution for b/316921158
Original change's description:
> Remove ability to use_goma in autoninja (chromeos excepted)
>
> Won't run a build if use_goma=true and target_os!="chromeos".
>
> Bug: b/277197166
> Test: Manual tests and updates to unit tests.
> Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166
Change-Id: Iaf506a10dfdc1c1672b2927636adf5773edc67c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5333176
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Won't run a build if use_goma=true and target_os!="chromeos".
Bug: b/277197166
Test: Manual tests and updates to unit tests.
Change-Id: I753618e6ad11cb623d9926a4af00a07339c02c43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5329167
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
Providing --package flag allows recipes.py to skip git calls it makes
to find repo root.
Bug: b/323243527
Change-Id: If507fbf15459d7ea73670df202e16c344f9467a8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5263441
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This will be used in cases where git workspace is unavailable.
Bug: b/323243527
Change-Id: I51056cfbac50f76a0de1b98cfec488d8f133e353
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5263021
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This reverts commit 1592a89c9f.
Reason for revert:
b/325856422
Original change's description:
> Remove ability to use_goma via autoninja.
>
> Doesn't run a build if use_goma is true, point to reclient docs. Turns
> off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
>
> Bug: b/277197166
> Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
> Reviewed-by: Ben Segall <bentekkie@google.com>
> Reviewed-by: Ramy Medhat <abdelaal@google.com>
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Commit-Queue: Michael Savigny <msavigny@google.com>
Bug: b/277197166 b/325856422
Change-Id: Iaec83b254af002f420bfd2df1154ce94e18fac15
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5308674
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Owners-Override: David Munro <davidmunro@google.com>
Reviewed-by: David Munro <davidmunro@google.com>
Use the log directory for the hash used to determime the socket used, so
developers get a distinct socket per build instead of per build
directory. Should allow concurrent builds against the same output
directory as much as ninja allows it.
Bug:b/321554715
Change-Id: I714dd2d32e2c651075055134e1568423a43eabdb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5290983
Auto-Submit: Michael Savigny <msavigny@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
Reviewed-by: Ben Segall <bentekkie@google.com>
git cl upload prompts a user if they upload a particularly large change,
determined by the number of CCs added on the change. With the watchlists
analyzer and eventual removal of CCing watchers from git cl, this check
needs to be updated to work without knowing the number of CCs.
This change does two things:
1. Moves the check from git cl upload to a pan-project presubmit check.
2. Update the heuristic of a LSC from > 100 CCs to > 100 files modified.
Bug: 324562917
Change-Id: I6bd0db9fde942273acebc320c8b52e81b02bd42f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5296199
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This is a reland of commit 4d2864f3a1
The previous change initiated "self._diff_cache()" per file.
Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: b/323243527
Change-Id: I4733b6870935d4200cb32255214c57cdb55e84f0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286636
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Had some debugging logging in autoninjas test. Removed it. Lines were erroneously added in https://crrev.com/c/5272474 to address b/277197166
Lines initially added to debug a problem in during development. In the initial review I received a +1 CR with a comment requesting they be removed. I made the change, started the patchset upload, went to a meeting, forgot to enter the patchset comment (thinking I had done it before the meeting) and submitted the change without the patchset. Basically context switching got me 😅
Bug: b/277197166
Change-Id: I504569c713dbca7302988c39230caf9e9e6fa2c0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5292094
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Michael Savigny <msavigny@google.com>
Doesn't run a build if use_goma is true, point to reclient docs. Turns
off use_goma for linux and mac builds. Windows will follow in 2-3 weeks.
Bug: b/277197166
Change-Id: Icf000ee5e4bdfeba82cadbbf2e02e7cd728e89b0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272474
Reviewed-by: Ben Segall <bentekkie@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Michael Savigny <msavigny@google.com>
This reverts commit 3569608028.
Reason for revert: This includes a fix for crbug.com/324358728.
The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.
[1] https://codereview.chromium.org/228353003
Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit e98e458a9f.
Reason for revert: keep depot_tools CCing watchers while b/324466185 is open
Original change's description:
> Deprecate --bypass-watchlists flag
>
> CCing watchers is now handled by the Watchlists AyeAye analyzer on
> all Gerrit projects that use it, and not CCing watchers locally has
> no effect. Rely on that instead of logic in git cl.
>
> Bug: 41494851, 40242793
> Change-Id: I58fdf8c0a741a9189cccec3b3b56d0a8a1c44058
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5276660
> Reviewed-by: Scott Lee <ddoman@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: 41494851, 40242793
Change-Id: Ied75beb163b2628343d4dcac5819b5bd77854de8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5281700
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Gavin Mak <gavinmak@google.com>
This reverts commit 4d2864f3a1.
Reason for revert: This still doesn't scale well with large commit:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8756578662173359185/+/u/presubmit/stdout?format=raw
This is now a different instruction called repeatedly needlessly:
```
git -c core.quotePath=false diff -p --no-color --no-prefix --no-ext-diff f0b151b0e519ceb3efae2ad8951d8a9ce86e4fbd...HEAD --no-renames; cwd=D:\b\s\w\ir\cache\builder\win_presubmit\src
```
called for each affected files.
We probably want to cache the output for this too?
Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
Bug: b/323243527, b/324293047
Change-Id: Iffde46facefa9d2e3a0d3a9ab5f8e753c53ea6d4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272317
Auto-Submit: Arthur Sonzogni <arthursonzogni@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This is a reland of commit dd1a596c3e
It is not a pure reland and adds support for caching submodules.
Original change's description:
> Refactor git functionality out of Change and _DiffCache
>
> A followup change will add support for change diff provided as user
> input through stdin/file.
>
> Bug: b/323243527
> Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: b/323243527
Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This reverts commit 3edda8d185.
Reason for revert: Breaks rebase-update; crbug.com/324358728
Original change's description:
> Update gclient to use git config caching
>
> This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
>
> This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
>
> A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
>
> Benchmarking
> ============
> With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
>
> Windows Benchmark
> =================
> Baseline (gpaste/6360045736951808): ~1min 12 sec.
> With Caching (gpaste/6480065209040896): ~1min 3sec.
> ~12.5% decrease in gclient sync noop runtime.
>
> Linux Benchmark
> ===============
> Baseline (gpaste/4730436763254784): ~3.739 sec.
> With Caching (gpaste/4849870978940928): ~3.534 sec.
> ~5.5% decrease in gclient sync noop runtime.
>
> Bug: 1501984
> Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: 1501984
Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
CCing watchers is now handled by the Watchlists AyeAye analyzer on
all Gerrit projects that use it, and not CCing watchers locally has
no effect. Rely on that instead of logic in git cl.
Bug: 41494851, 40242793
Change-Id: I58fdf8c0a741a9189cccec3b3b56d0a8a1c44058
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5276660
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This reverts commit dd1a596c3e.
Reason for revert: This doesn't scale well with large commit:
https://issues.chromium.org/issues/324293047
Original change's description:
> Refactor git functionality out of Change and _DiffCache
>
> A followup change will add support for change diff provided as user
> input through stdin/file.
>
> Bug: b/323243527
> Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: b/323243527
Change-Id: Ic5179de6744a2ead629f512a5098f6e8c63d01c8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5277255
Commit-Queue: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Arthur Sonzogni <arthursonzogni@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This removes logspam from `git cl` when a user (like me) hasn't opted
either in or out. DOGFOOD_STACKED_CHANGES=1 is the codified behavior.
This removes the DOGFOOD_STACKED_CHANGES=0 opt-out.
Bug: b/265929888
Change-Id: Ia159f898512d815e3a4b76cc8859c599016a7bec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5274266
Auto-Submit: Peter Boström <pbos@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.
Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.
Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
A followup change will add support for change diff provided as user
input through stdin/file.
Bug: b/323243527
Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
git_cache populate can get interrupted midway (e.g. by LUCI CV sending a
signal that build is no longer needed). When that happens, a git mirror
may be in a state where some commits are available, but cloning such
repositry results in an empty repsitory.
This change ensures that the initial fetch operation finished
successfully.
R=ddoman@google.com
Bug: 1517944
Change-Id: I1ee860860877dbfff7a444b45fe4515fe26b248c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5219032
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
+ Change scm GetAllFiles back to returning all files
+ should be safe to do since it's only used by presubmit code
+ add ignore_submodules option to CaptureStatus
+ It's used by other parts of depot_tools code
+ Add AffectedSubmodules to return submodules only
+ AffectedFiles still only returns non-submodule files.
Bug: 1475770
Change-Id: I457ad96099fb20eff4cc1a4fc84f59e7ae707abe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5216005
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
The previous binary was so old that it does not work on recent
versions of macOS.
Note that `cipd selfupdate-roll` requires the cipd package to exist
at the given tag for all platforms that the package has ever been
built for. Therefore, I'm using a revision after the latest platform
was added to the builders.
Change-Id: I865c31a5e3823ffc344bcb6585a29935ef6214a6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5191588
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Brian Ryner <bryner@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
The cache won't be used if ResolveCommit throws an exception or when
non-sha1 is used.
This optimization saves about 7% of overall gclient sync on
chromium/src.
R=gavinmak@google.com
Change-Id: I9c36d937dc7b8553818888e2e9c4eecd029a978c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5147496
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Assigning os.environ the result of os.environ.copy() replaces it
with a standard dict. This breaks reflection into the actual
environment, as well as features such as case-insensitivity on
Windows, leading to undefined standard library behavior.
R=bryner, gavinmak
Bug: 1511316
Change-Id: I63d026387104ae92669256567854f8bf9d2397a2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5133887
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is to remove 1~3 seconds from no-op build using autoninja.
Bug: b/309720176
Change-Id: I2c15a43517d5a99eae794015c18c884a3f3fdef4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5092847
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
This change updates the `_CheckClean()` fn to use `git status` instead
of `git update-index` and `git diff-index`. The `_CheckClean()` fn is
run during every update and this change reduces the subprocess calls
made by it by half.
Change-Id: Ie9a23b0bce748bec4cac88df09655569e88e4841
Bug: 1501984
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5076224
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
From my recent go/gclient-sync-no-ops analysis, I noticed that
HostArch() is repeated called within get_builtin_vars() fn. The
`platform.architecture()` call made in arm-based systems and some x64 is
expensive and adds to gclient sync's runtime when repeatedly called.
When caching HostArch(), I noticed 22.4% speed-up in incremental no-op
gclient sync when run on arm-based machine.
Change-Id: I962c4fb0879d2931268f5bec1678ef782c56e7f2
Bug: 5076224
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5076226
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This disallows Googler to use non-google account with reclient and
siso from corp machine.
Bug: b/309720176
Change-Id: I8077eff8bbf47f579462e1fefa3609a5d492a013
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5035266
Reviewed-by: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
The code rotating the old build logs expects to delete directories
(since it uses rmtree) but tries to delete all items returned by
os.listdir(...) which may includes symbolic links or regular files.
Change it to only consider the old build logs directories instead.
Based on crrev/5028323, with fix to unit test.
Fixed: b/310900283
Change-Id: I4c618e6618c0193331c063028ebf02d8c4e7baee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5038467
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Ramy Medhat <abdelaal@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
The package is deprecated and will be removed in py 3.12.
R=ddoman@google.com
Bug: 1500415
Change-Id: Ia130c556ad3f2135fac4a9b1d9bb4ecc088cfe9d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5030818
Reviewed-by: Scott Lee <ddoman@chromium.org>
Reviewed-by: Jonathan Nieder <jrn@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Commit-Queue: Jonathan Nieder <jrn@google.com>
.git suffix is not well supported by the systems that utilize
submodules. Since some tools expect DEPS to have .git suffix, it's not
possible to simply update DEPS without breaking the tools.
This patch updates `gclient gitmodules` to omit .git suffix and trailing
slashes. However, if `.gitmodule` already exists, we are expecting the
tool to generate the same result if DEPS didn't change.
R=aravindvasudev@google.com
Bug: b/303296048
Change-Id: I30497b9d5f4e9681e16b4a913151de0f4a7c9545
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5008942
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
On unsupported platforms, luci-auth is not downloaded from CIPD and
hence the scripts relying on gsutil.py fail. Given we aren't using
luci-auth in unsupported platforms anyway, skip wrapping gsutil with
luci-auth on unsupported platforms.
Curently, only "aix" is added to the unsupported platforms list
eventhough we have other unsupported platforms too. This list will be
expanded if necessary.
R=yiwzhang
Change-Id: I70f6fceddc672c76bbea9a108a75b84662e26459
Fixed: 1467752
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4989412
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Instead of printing a command-line, we just directly call into the respective main functions from Python. This saves spawning another interpreter and prevents things that can go wrong from having to quote, unquote, split and tunnel arguments through shells.
Part of my bigger auto{ninja,siso} refactoring.
Tested:
- Handling of the ^^ suffix on Windows still works correctly.
- Handling of error codes - i.e.; making sure
"autoninja base_unittests && base_unittests.exe" behaves properly
in the success/failure case.
- Make sure the command prompt title is reliably reset on exit.
I tested autoninja with all combinations of these:
- Host platform: Linux, macOS, Windows
- Remote GN args: <none>, use_goma=true, use_remoteexec=true
- Siso GN args: <none>, use_siso=true
- Targets: base, ../../base/types/expected_macros_unittest.cc^ (on Linux) and ../../base/types/expected_macros_unittest.cc^^ (on Windows)
R=brucedawson@chromium.org, jwata@google.com, tikuta@chromium.org
Bug: b/293657720
Change-Id: I275a775fdc5abb6555f79d4beab76cd0914d4bd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4924185
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Josip Sokcevic
Takuto Ikuta
Yiwei Zhang
Scott Lee
Gavin Mak
Alexander Schulze
Stephanie Kim
Joanna Wang
Aravind Vasudevan
Michael Savigny
Jinli Wu
Robert Iannucci
Michael Ershov
mlcui
Arthur Sonzogni
Arthur Sonzogni
Peter Boström
Brian Ryner
Helmut Januschka
Ben Pastene
Ben Segall
Philipp Wollermann