It worked only because of an all-powerful service account on each bot,
which isn't the case any more.
R=sokcevic@google.com
Change-Id: Id66f9cc44cd416e5b61745c0f690d5a91b3b4c67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2225976
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This is a reland of d3affaa624
Original change's description:
> Use OS level locking in git_cache.py
>
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
>
> R=apolito@google.com, ehmaldonado@chromium.org
>
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1049610
Change-Id: I58e65a10f7c779e0de1121ba7167c694996e390c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211189
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit d3affaa624.
Reason for revert: no attribute ignore_lock
Original change's description:
> Use OS level locking in git_cache.py
>
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
>
> R=apolito@google.com, ehmaldonado@chromium.org
>
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
TBR=iannucci@chromium.org,ehmaldonado@chromium.org,apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com
Change-Id: Iecc963e0a99d7f59f3f8801e529839346f9fbaf3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1049610
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211186
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Without OS level locking it's possible to leave "lock" files on disk
which will prevent next run to acquire those locks. This can easily
happen if SIGKIL is issued.
R=apolito@google.com, ehmaldonado@chromium.org
Bug: 1049610
Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
If a local branch tracks removed remote tracking branch (e.g. foo) and
remote has a branch that starts with such name (e.g. foo/bar), git fetch
will fail. --prune flag removes any remote-tracking branches that no
longer exist.
R=apolito@google.com, ehmaldonado@chromium.org
Bug: 1079483
Change-Id: I9bc31bf961d52a86b6fa2342249971b99a003666
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2190341
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.
Change-Id: I71689e30bdede392588c69e118e9297d86a134a3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2120281
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 32a801b5da.
Reason for revert:
It's causing depot tools presubmit to time out
(Also, introduces a Python 3 error for git_cache).
Original change's description:
> [bot_update] Also bootstrap in the case that there are 0 pack files
>
> This can happen if the cache repo was init'd, but has no pack files;
> previously it would try to fetch from scratch.
>
> R=ehmaldonado@chromium.org, tandrii@chromium.org
>
> No-Presubmit: true
> Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
TBR=iannucci@chromium.org,tandrii@google.com,ehmaldonado@chromium.org
Change-Id: I93366bcc6ff1df579a67d51c2cda812c84a06215
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1951985
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.
R=ehmaldonado@chromium.org, tandrii@chromium.org
No-Presubmit: true
Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
I noticed that chromium/src archive grew 2x from 7.3GiB in August
to 15+ GiB now. Turns out --aggressive-gc had no effect.
R=iannucci
Change-Id: I6a27f67f5a4dd2101ac1622221106e094af4688a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1928583
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Running gclient_smoketests times out on windows when using git daemon,
so use a local directory as remote instead.
Bug: 1024683
Change-Id: I6ca506d74de463d914317f176eefbe74996298c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1879723
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
There are 3 layers modified, starting from the bottom up:
1. git_cache.py populate
Now takes a --no-fetch-tags option. If specified, the cache will not
fetch updated tags from the server by passing --no-tags to git fetch.
This prevents the server from sending all the tag refs. In chromium
this prevents significant time bottlenecks dealing with 10k+ tags.
2. bot_update.py
bot_update has to deal with multiple git repos, it has the root repo
that is checked out through git-cache, as well as repos synched via
DEPS and gclient.
The script now takes a --no_fetch_tags option. If specified,
the git-cache populate call will include --no-fetch-tags. Otherwise, it
won't. So this controls (for chromium) if fetches to the src.git server
are passed --no-tags.
3. bot_update/api.py
This is the entry point for recipes and also has to deal with multiple
git repos. The behaviour at this point is not changed from the default.
A |no_fetch_tags| parameter is added to ensure_checkout() that defaults
to False.
This CL is a refactor with no intended behavior change.
The next step will be to change the chromium build repo to pass True
for ensure_checkout(no_fetch_tags) on chromium trybots.
This represents solution #2 in
https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit#
Bug: 1019824
Change-Id: I935430603299daa9e301a95a5184c0ce486fd912
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
This reverts commit 1e4dbf3f64.
Reason for revert: unlikely but possible cause of http://shortn/_sgL4PqICVB
Original change's description:
> Add more git tracing.
>
> This CL adds more perf tracing to `git fetch` from bot_update.py and
> git_cache.py
>
> Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: infra
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Auto-Submit: Erik Chen <erikchen@chromium.org>
TBR=danakj@chromium.org,tandrii@google.com,erikchen@chromium.org
Change-Id: I03b33bf1532212f83a6b89b1de32fafb6b8aafc4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891001
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
This CL adds more perf tracing to `git fetch` from bot_update.py and
git_cache.py
Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Auto-Submit: Erik Chen <erikchen@chromium.org>
We currently fetch tags, which consumes ~17k lines. This limit ensures that we
won't blow up the log files in exceptional circumstances [currently setting
around ~2MB], but still allows us to get all the logs.
Change-Id: Ib690aaa07e2bde8549d221b90511b6c4863c3358
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: skia
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1867971
Commit-Queue: Erik Chen <erikchen@chromium.org>
Auto-Submit: Erik Chen <erikchen@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
This will allow us to separate the 'populate' and 'upload' phases
in the git cache updater recipe, which will allow us to do all the
'populate' steps in parallel, but then limit the parallelism of the
`pack/gc/upload` operations.
R=tandrii@chromium.org
Change-Id: I8b8a9155f86350be37ed5a67c592ff1fec4d42ff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1830857
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
I know that sometimes imports can have side-effects,
so unused imports shouldn't always be removed, but these
ones look like they could be.
Change-Id: Iea9f82afa99b0ea35f29a28f20ce0493b579cfee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1819860
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
This reverts commit 82ae4b4b7d.
Reason for revert: didn't help. Something else limits 'git gc' by 2GB.
However, even so, total amount of .pack files for chromium/src is <7.5GB.
Original change's description:
> git-cache: don't limit pack files to 2GiB.
>
> This is only relevant to 1 builder which updates git cache bundles.
> Since some time in 2018-2019, it started creating several 2GiB packs
> for chromium/src. I verified that the setting affecting it is
> core.deltaBaseCacheLimit
>
> Log file of a build which I observed while ssd-ed into the machine (internal):
> https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8901682638839779824/+/steps/Updating_https:__chromium.googlesource.com_chromium_src/0/stdout
> Git version: version:2.21.0.chromium16
>
> R=ehmaldonado
>
> Change-Id: I52cadfb9f34faea09a57d53387cab7e0538362b9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1818076
> Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
TBR=tandrii@google.com,ehmaldonado@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I24915555c703bc92fda921524820405de5d69f7d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1817101
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This will be used on internal cache updater.
For instance, I've just compressed chromium/src resulting bootstrap
files from 20GiB to 7.5 GiB.
R=ehmaldonado
Change-Id: I15411700eb2ac3a26d1c658a12288cc49e48fd48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1802877
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
The only upside of fetching everything directly from GoB
is that git cache would get just 1 .pack. However, git cache
already runs "git gc" before uploading cache to GS, so
this point is moot.
R=maruel
Bug: 943696
Change-Id: Ie8e77a81aa81489dae240b7c767c5842a12c6f19
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1585641
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
We should only be maintaining one cache bundle per repo, but it turns
out that we've had two in the past due to GoB supporting two different
paths to the repo, and users were getting stale bundles as a result.
This CL fixes things so that we should only get a single bundle per
repo.
Bug: 935084
Change-Id: I0d6713280a2abbc20e35ff87e7be115870dd5140
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1566431
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
This enables gclient sync and gclient runhooks to run, barring hook script failures.
git cl upload also now works.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: I2ac587b5f803ba7f5bb5e412337ce049f4b1a741
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524583
Commit-Queue: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
We have trouble rolling dep of the grpc library. It looks like buildbots
use git_cache to download cache of the library from cloud storage, but
the way it picks up the latest cache is to do a string sort on the
filenames then pick the last one. This won't work if the filenames have
digit carrying, say you have both 9999.zip and 10000.zip, then 9999.zip
will get picked up.
This CL fixes this by implementing a new filename sorting logic that
extracts the numeral part of the filename and sort on it.
Bug: 927154
Change-Id: I68fce3fe67e55ce5092e7e9dc1dca606b427fe87
Reviewed-on: https://chromium-review.googlesource.com/c/1448954
Commit-Queue: Yuwei Huang <yuweih@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Modern macOS has no problem unzipping big zip files. To test:
$ gsutil cp gs://chromium-git-cache/chromium.googlesource.com-chromium-src/516491.zip test.zip
$ unzip -l test.zip
Change-Id: I84b3cff21cb9b7033c04b427e23f27a75ab1d8ae
Reviewed-on: https://chromium-review.googlesource.com/1152294
Commit-Queue: Jeremy Apthorp <jeremya@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
The update_scripts step doesn't set '--reset'
Additionally, if there was no fetch config, don't fail trying to unset it.
Tbr: agable@chromium.org
No-Try: True
Bug: 862547
Change-Id: I90886ca7d1dd20ae59b378a5998de47dc67c60f9
Reviewed-on: https://chromium-review.googlesource.com/1137693
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This CL makes a couple changes:
* The goal here is to be able to specify git cache entirely from the
environment variable $GIT_CACHE_PATH and not require special treatment from
bot_update. Eventually this will be specified at the swarming task level
instead of in the recipe (i.e. "cached git" will eventually be an
implementation detail of git on the bots and completely transparent to
all other software).
* Removal of the general --cache-dir option from gclient. This option was
error-prone; it doesn't actually make sense to configure this on
a per-invocation basis. The sole exception was `gclient config`, which
now allows this option to be set directly.
* Consolidation of GitWrapper.cache_dir and GitWrapper._GetMirror; previously
these two things could disagree, leading to weird intermediate states. Now
they're the same value.
R=agable@chromium.org, ehmaldonado@chromium.org, tandrii@chromium.org
Bug: 823434
Change-Id: I14adc7619b5fc10768ce32be2651c6215ba94aff
Reviewed-on: https://chromium-review.googlesource.com/1105473
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
This partially reverts commit d51ed57edb.
Reason for revert:
New git client for windows was rolled including fix for slow `git fetch`.
I guess smaller pack limit causes frequent bootstrap taking 2~3 minutes longer than
the case it does not happen.
Let me see what happen if we increase pack limit 9 -> 30.
I will increase this to 50 if this won't cause regression again.
Original change's description:
> git_cache: lower max num of .pack files before re-bootstrap on Win.
>
> It used to be 50, I think ~9 gives best results for Chromium on Win:
> on golo VM, it takes <4 minutes to re-boostrap + git fetch small
> delta, assuming zipped git checkout for bootstrap is fresh (~1day).
>
> For other repos, which are significantly smaller, this change should
> have minor effect if at all.
>
> Test: I tested this using `led` tool on Win7 machines running LUCI
> stack extensively. For example,
>
> * https://ci.chromium.org/swarming/task/3a0102e8c8657410
> shows case with few .pack files, hence just 1 fetch
>
> * https://ci.chromium.org/swarming/task/3a010282f9fd8010
> shows case with 39 .pack files and so bootstrapping + fetch.
> If you look at prior tasks on the same VM, you'd find this:
> https://ci.chromium.org/swarming/task/39ffe843d01ed010
> which spent 8 minutes doing 1 incremental fetch with 39 .pack
> files.
>
> **Troopers/Sheriffs**: This change is safe to revert.
> However, beware that you should also at the same time revert the recipe
> roll of this CL to the repo, in which the failed builder's recipe is
> located, typically `chromium/tools/build`.
>
> Bug: 749709
> Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
> Reviewed-on: https://chromium-review.googlesource.com/787174
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 749709
Change-Id: I3052abe4a9b53277a60c0791a85355e7a0bbdf8f
Reviewed-on: https://chromium-review.googlesource.com/823544
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
Tested end-to-end, for example
https://ci.chromium.org/swarming/task/3a0147207378b910
which contains:
src/media/cdm/api (Elapsed: 0:00:01)
----------------------------------------
[0:00:00] Started.
_____ src\media\cdm\api at ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237
[0:00:00] running "git cat-file -e ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237" in "e:\b\s\w\ir\cache\git\chromium.googlesource.com-chromium-cdm"
skipping mirror update, it has rev=ea5df8e78fbd0a4c24cc3a1f3faefefcd1b45237 already
thereby saving on needless git fetch (~40s in glcient sync on win trybots),
and reducing the rate of .pack file accumulation inside cache directories.
Risks: silently broken recipes which run gclient sync (or worse, bot_update)
as a means of fetching latest commits in all repos of a solution. I think
the benefit of faster bot_update in chromium CQ is worth the potential risk.
PSA: https://groups.google.com/a/chromium.org/d/msg/infra-dev/UYLdBwAXm1Y/OV9QB6JnBQAJ
Bug: 749709
Change-Id: I7a9e8ab82a5e2b848e450f19a798ac18a0b5e201
Reviewed-on: https://chromium-review.googlesource.com/787331
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
It used to be 50, I think ~9 gives best results for Chromium on Win:
on golo VM, it takes <4 minutes to re-boostrap + git fetch small
delta, assuming zipped git checkout for bootstrap is fresh (~1day).
For other repos, which are significantly smaller, this change should
have minor effect if at all.
Test: I tested this using `led` tool on Win7 machines running LUCI
stack extensively. For example,
* https://ci.chromium.org/swarming/task/3a0102e8c8657410
shows case with few .pack files, hence just 1 fetch
* https://ci.chromium.org/swarming/task/3a010282f9fd8010
shows case with 39 .pack files and so bootstrapping + fetch.
If you look at prior tasks on the same VM, you'd find this:
https://ci.chromium.org/swarming/task/39ffe843d01ed010
which spent 8 minutes doing 1 incremental fetch with 39 .pack
files.
**Troopers/Sheriffs**: This change is safe to revert.
However, beware that you should also at the same time revert the recipe
roll of this CL to the repo, in which the failed builder's recipe is
located, typically `chromium/tools/build`.
Bug: 749709
Change-Id: I18e2b63283100d466e9fb981a9094862463f6909
Reviewed-on: https://chromium-review.googlesource.com/787174
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
This is short of sending this metrics to ts_mon, but is still useful
for diagnose and then claim improvements in git cache performance on
bots.
This change has been tested for realz with led, particularly on windows
https://ci.chromium.org/swarming/task/3a01358af14a7d10
Bug: 749709
Change-Id: I2b3589079d2caa7f70007f90fcbce85a0205d24b
Reviewed-on: https://chromium-review.googlesource.com/787173
Reviewed-by: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
These aren't in use, and the original problem they were
meant to solve has been solved at the gclient.py layer
using resource locking:
https://codereview.chromium.org/2049583003
Bug: 773008
Change-Id: I6609f39d7f15604e0bb3d742a41c4f9fec87a57a
Reviewed-on: https://chromium-review.googlesource.com/707728
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Also - Turn on GC for the unrecognized repos.
BUG=skia:6241
TEST=local
git_cache.py populate with chromium and non-chromium repo.
Change-Id: I100d3665e317b29c5b3f69b49c165ce88cd019d6
Reviewed-on: https://chromium-review.googlesource.com/455979
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This affects a bunch of files, but only changes comments,
and shouldn't make any difference to behavior.
The purpose is to slightly improve readability of pylint
disable comments.
Change-Id: Ic6cd0f8de792b31d91c6125f6da2616450b30f11
Reviewed-on: https://chromium-review.googlesource.com/420412
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Currently, 'bot_update' uses the 'gclient' that is on the system path.
Now, it will use the 'gclient.py' that is in the same 'depot_tools'
checkout as the 'bot_update' recipe module.
Also don't ignore "git_cache" move errors.
BUG=664254,663990,663440
TEST=None
Review-Url: https://codereview.chromium.org/2492963002
gclient-sync + git-cache are racy because if two different DEPS files
dependent on the same repo URL, even if depsed to different checkout
dirs, they use same git cache dir. If both checkout dirs are synced at
the same time, one of them cannot acquire a lock.
This is a cheap solution to change --lock_timeout option default value
from 0 to 5m.
R=hinoka@chromium.org
BUG=593468
Review URL: https://codereview.chromium.org/1785083005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@299386 0039d316-1c4b-4281-b951-d872f2087c98
I guess I'm the only developer using git-cache, which is sad.
Hopefully these fixes will make it easier to adapt this to developer
usage some time in the FUTURE.
BUG=583420
TEST=Works for me
R=agable@chromium.org,tandrii@chromium.org
Review URL: https://codereview.chromium.org/1650993005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298531 0039d316-1c4b-4281-b951-d872f2087c98
With this change, 'git cache fetch' will automatically re-bootstrap
a repo from google storage if the local cache has too many pack
files. The behavior can be disabled with the --no-bootstrap flag.
BUG=
Review URL: https://codereview.chromium.org/1320383004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296829 0039d316-1c4b-4281-b951-d872f2087c98
Based on yapf (https://github.com/google/yapf) this
formatter currently only works with --full. It defaults
to pep8 style and projects that use a different style
can add .style.yapf to the top level.
Review URL: https://codereview.chromium.org/1156743008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295547 0039d316-1c4b-4281-b951-d872f2087c98
Handle KeyboardInterrupt gracefully rather the printing a
backtrace. Most users of these tools don't expect a
backtrace when then hit Ctrl-C.
Also, fix a few other inconsistencies found in the python
startup code of these different scripts:
- always call main function 'main' (rather than 'Main')
- always return 0 from main function
- if main takes args never include argv[0]
Review URL: https://codereview.chromium.org/955993006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294250 0039d316-1c4b-4281-b951-d872f2087c98
The root of problem is a _cache_temp file.
git_cache expected that it is a folder.
So rmtree failed to remove it.
BUG=
TBR= dpranke@chromium.org, enne@chromium.org
NOTRY=true
Review URL: https://codereview.chromium.org/825133002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293502 0039d316-1c4b-4281-b951-d872f2087c98
This pins gsutil to a vanilla 4.7 instead of the weird custom 3.4 we have in depot_tools
BUG=
R=pgervais@chromium.org
Review URL: https://codereview.chromium.org/797663003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293413 0039d316-1c4b-4281-b951-d872f2087c98
So the original intention of moving it to a different directory to be deleted later
was to (1) save it for diagnosis (2) be a single inode swap rather than a long
rmtree delete.
1. No one is actually looking at these directories
2. The tmpdir sometimes end up on a different partition, so it ends up being
a copy + delete instead. Since we're not getting timeouts from that, its
probably actually better to just straight up delete it.
BUG=410727
Review URL: https://codereview.chromium.org/538993002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@291818 0039d316-1c4b-4281-b951-d872f2087c98
Seems like some combination of things (bad cache dir, shallow checkout, etc) causes
git cache to not clean up properly before trying to move new directory A into
already existing directory B.
So the logic is all there to detect that the cache is invalid, its just not
being cleaned up properly during shallow checkouts.
BUG=406864
TBR=iannucci
Review URL: https://codereview.chromium.org/499123002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@291587 0039d316-1c4b-4281-b951-d872f2087c98
Some options have words separated by underscores. Added options with
same name and underscores replaced by hyphens.
BUG=400953
Review URL: https://codereview.chromium.org/436963005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@288366 0039d316-1c4b-4281-b951-d872f2087c98
If you're in a git checkout cloned from the git cache, this will:
- Update the cache with the latest upstream commits.
- Update the cwd with the latest commits from the cache.
For example:
> cd $HOME/workspace/chromium/src
> git cache fetch
R=agable@chromium.org,iannucci@chromium.org
BUG=
Review URL: https://codereview.chromium.org/440213005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@288140 0039d316-1c4b-4281-b951-d872f2087c98
Git cache sometimes fail on:
Traceback (most recent call last):
File "E:\b\depot_tools\\gclient.py", line 2064, in <module>
sys.exit(Main(sys.argv[1:]))
File "E:\b\depot_tools\\gclient.py", line 2052, in Main
return dispatcher.execute(OptionParser(), argv)
File "E:\b\depot_tools\subcommand.py", line 245, in execute
return command(parser, args[1:])
File "E:\b\depot_tools\\gclient.py", line 1830, in CMDsync
ret = client.RunOnDeps('update', args)
File "E:\b\depot_tools\\gclient.py", line 1342, in RunOnDeps
work_queue.flush(revision_overrides, command, args, options=self._options)
File "E:\b\depot_tools\gclient_utils.py", line 852, in flush
self._run_one_task(self.queued.pop(i), args, kwargs)
File "E:\b\depot_tools\gclient_utils.py", line 944, in _run_one_task
task_item.run(*args, **kwargs)
File "E:\b\depot_tools\\gclient.py", line 744, in run
file_list)
File "E:\b\depot_tools\gclient_scm.py", line 160, in RunCommand
return getattr(self, command)(options, args, file_list)
File "E:\b\depot_tools\gclient_scm.py", line 387, in update
self._UpdateMirror(mirror, options)
File "E:\b\depot_tools\gclient_scm.py", line 802, in _UpdateMirror
ignore_lock=options.ignore_locks)
File "E:\b\depot_tools\git_cache.py", line 409, in populate
os.rename(tempdir, self.mirror_path)
WindowsError: [Error 183] Cannot create a file when that file already exists
It would appear that its being racy, but otherwise it doesn't make any sense. A theory
is that this could be running twice and stepping on each other. Allowing this
to pass on OSError allows us to test this theory.
BUG=395333
Review URL: https://codereview.chromium.org/408653002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@284272 0039d316-1c4b-4281-b951-d872f2087c98
We're seeing fetches fail in interesting ways:
running "git fetch -v --progress origin +refs/heads/*:refs/heads/*" in "/mnt/scratch0/b_used/build/slave/cache_dir/chrome--internal.googlesource.com-chrome-src--internal"
error: object file ./objects/a1/4bd89aa4cc7d7bbad7594cba0ae73e99dbb54c is empty
error: object file ./objects/a1/4bd89aa4cc7d7bbad7594cba0ae73e99dbb54c is empty
fatal: loose object a14bd89aa4cc7d7bbad7594cba0ae73e99dbb54c (stored in ./objects/a1/4bd89aa4cc7d7bbad7594cba0ae73e99dbb54c) is corrupt
fatal: The remote end hung up unexpectedly
And then the cache becomes corrupted. This blows the cache away if this happens.
BUG=261741
Review URL: https://codereview.chromium.org/352543003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@280123 0039d316-1c4b-4281-b951-d872f2087c98
Unfortunately, the locking mechanism is still flaky on Windows.
bot_update.py will use this, since we can be certain that there
won't be overlapping access to the cache.
R=hinoka@google.com, agable@chromium.org, hinoka@chromium.org
BUG=
Review URL: https://codereview.chromium.org/340953003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@278490 0039d316-1c4b-4281-b951-d872f2087c98
A number of bots have been running out of disk space because they
are accumulating temp dirs and temp pack files. Be more careful
to clean up temp dirs whenever UnlockAll() is called.
R=hinoka@chromium.org,agable@chromium.org
BUG=
Review URL: https://codereview.chromium.org/335253002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@277812 0039d316-1c4b-4281-b951-d872f2087c98
When a git-cache operation is interrupted, it can leave behind large
temporary pack files. Over time, these pack files will fill up
disks.
R=agable@chromium.org,hinoka@chromium.org,iannucci@chromium.org
BUG=352268
Review URL: https://codereview.chromium.org/326203003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@276208 0039d316-1c4b-4281-b951-d872f2087c98
Last patch (https://codereview.chromium.org/313933005/) forgot to take into account using a boto file (setting boto to os.devnull
means connect unauthenticated, setting it to None will allow it to do the default
behavior)
This patch will always use a boto file, which should be fine.
BUG=261741
Review URL: https://codereview.chromium.org/314203002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@275230 0039d316-1c4b-4281-b951-d872f2087c98
This is need, e.g., to suppress logging message for 'exists'.
TBR=agable@chromium.org,hinoka@chromium.org
BUG=
Review URL: https://codereview.chromium.org/299163002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@272450 0039d316-1c4b-4281-b951-d872f2087c98
This is because the version of unzip on osx doesn't support zip64, so it can't
unzip large repositories like blink.
BUG=261741
Review URL: https://codereview.chromium.org/251413002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@266151 0039d316-1c4b-4281-b951-d872f2087c98
Sometimes, removing lockfiles fails on windows, for obscure reasons.
Following advice from gclient_utils.rmtree, lock file removal has
been improved by using the builtin executable 'del' and retrying 3
times in case it fails.
BUG=
Review URL: https://codereview.chromium.org/240653002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@264557 0039d316-1c4b-4281-b951-d872f2087c98
Michael Moss
Andrii Shyshkalov
Josip Sokcevic
Edward Lesmes
Edward Lesmes
Robert Iannucci
Andrii Shyshkalov
John Budorick
danakj
Erik Chen
Quinten Yearsley
Karen Qian
Dirk Pranke
Raul Tambre
Yuwei Huang
Jeremy Apthorp
Takuto Ikuta
Vadim Shtayura
Ryan Tseng
Quinten Yearsley
dnj
sergeyberezin
hinoka
nodir@chromium.org
szager@chromium.org
sbc@chromium.org
loislo@chromium.org
szager@chromium.org
pgervais@chromium.org
hinoka@chromium.org
hinoka@google.com