This reverts commit d3cfea0c78.
Reason for revert: using self.GSUTIL_CP_SEMAPHORE instead of
global GSUTIL_CP_SEMAPHORE
Original change's description:
> Limit number of concurrent gsutil cp calls
>
> gsutil utilizes as many cores as possible, and will create many threads
> too. In combination with gclient doing the same thing (utilize as many
> cores as possible), this results in creating way too many processes and
> some builders are running out of resources (e.g. can't create threads).
>
> This limits number of gsutil cp commands in git_cache to two.
>
> R=apolito@google.com, dpranke@google.com
>
> Bug: 1255228, 1249003
> Change-Id: Ifcccaacde6c9e1c60c1da0a0bc5a3e512e158dcc
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219900
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1255228, 1249003
Change-Id: I0fde2477ea620c683b49efaa2a70262bd8e615e2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3221900
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
gsutil utilizes as many cores as possible, and will create many threads
too. In combination with gclient doing the same thing (utilize as many
cores as possible), this results in creating way too many processes and
some builders are running out of resources (e.g. can't create threads).
This limits number of gsutil cp commands in git_cache to two.
R=apolito@google.com, dpranke@google.com
Bug: 1255228, 1249003
Change-Id: Ifcccaacde6c9e1c60c1da0a0bc5a3e512e158dcc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3219900
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Currently, git cache depends on the global default branch. Set it to
main explicitly instead.
Bug: 3082645, b/195975120
Change-Id: Ia9331d479dbe9547b6913c74bbf86852fff3a596
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3083251
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
This is a reland of 4c67f856f0
Issues should have been fixed by crrev.com/c/2838026
Original change's description:
> git-cache: Add option to fetch commits.
>
> Add option to git cache to fetch commits.
> And use it in bot_update and gclient sync to make sure
> the needed commits are present on the checkout.
>
> Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: Ie5a29737f5a75d28bc7c5c2f6cb99ec7f87cd9e8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2841046
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 4c67f856f0.
Reason for revert: crbug.com/1200657
Original change's description:
> git-cache: Add option to fetch commits.
>
> Add option to git cache to fetch commits.
> And use it in bot_update and gclient sync to make sure
> the needed commits are present on the checkout.
>
> Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: I31a1defb26119bcb46894ab4084d626b65c45fed
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2837413
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Add option to git cache to fetch commits.
And use it in bot_update and gclient sync to make sure
the needed commits are present on the checkout.
Change-Id: I9e90da9e3be6e7bacf714b22bf0b735463e655b6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2829942
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
logging.warn is deprecated and should be replaced with logging.warning.
Bug:1097402
Change-Id: I55ce007bec47608eeafcf3cd9ac2b90f60073115
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2575621
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
It worked only because of an all-powerful service account on each bot,
which isn't the case any more.
R=sokcevic@google.com
Change-Id: Id66f9cc44cd416e5b61745c0f690d5a91b3b4c67
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2225976
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This is a reland of d3affaa624
Original change's description:
> Use OS level locking in git_cache.py
>
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
>
> R=apolito@google.com, ehmaldonado@chromium.org
>
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1049610
Change-Id: I58e65a10f7c779e0de1121ba7167c694996e390c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211189
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit d3affaa624.
Reason for revert: no attribute ignore_lock
Original change's description:
> Use OS level locking in git_cache.py
>
> Without OS level locking it's possible to leave "lock" files on disk
> which will prevent next run to acquire those locks. This can easily
> happen if SIGKIL is issued.
>
> R=apolito@google.com, ehmaldonado@chromium.org
>
> Bug: 1049610
> Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
TBR=iannucci@chromium.org,ehmaldonado@chromium.org,apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com
Change-Id: Iecc963e0a99d7f59f3f8801e529839346f9fbaf3
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1049610
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2211186
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Without OS level locking it's possible to leave "lock" files on disk
which will prevent next run to acquire those locks. This can easily
happen if SIGKIL is issued.
R=apolito@google.com, ehmaldonado@chromium.org
Bug: 1049610
Change-Id: Id87aa1376b9ea5ff0c2d14f3603636493ed1dd5b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2189333
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
If a local branch tracks removed remote tracking branch (e.g. foo) and
remote has a branch that starts with such name (e.g. foo/bar), git fetch
will fail. --prune flag removes any remote-tracking branches that no
longer exist.
R=apolito@google.com, ehmaldonado@chromium.org
Bug: 1079483
Change-Id: I9bc31bf961d52a86b6fa2342249971b99a003666
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2190341
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.
Change-Id: I71689e30bdede392588c69e118e9297d86a134a3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2120281
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 32a801b5da.
Reason for revert:
It's causing depot tools presubmit to time out
(Also, introduces a Python 3 error for git_cache).
Original change's description:
> [bot_update] Also bootstrap in the case that there are 0 pack files
>
> This can happen if the cache repo was init'd, but has no pack files;
> previously it would try to fetch from scratch.
>
> R=ehmaldonado@chromium.org, tandrii@chromium.org
>
> No-Presubmit: true
> Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
TBR=iannucci@chromium.org,tandrii@google.com,ehmaldonado@chromium.org
Change-Id: I93366bcc6ff1df579a67d51c2cda812c84a06215
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1951985
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This can happen if the cache repo was init'd, but has no pack files;
previously it would try to fetch from scratch.
R=ehmaldonado@chromium.org, tandrii@chromium.org
No-Presubmit: true
Change-Id: I97e863dd6c2ecfd00fdb4238dda48e0d3929c4f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1941337
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
I noticed that chromium/src archive grew 2x from 7.3GiB in August
to 15+ GiB now. Turns out --aggressive-gc had no effect.
R=iannucci
Change-Id: I6a27f67f5a4dd2101ac1622221106e094af4688a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1928583
Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Running gclient_smoketests times out on windows when using git daemon,
so use a local directory as remote instead.
Bug: 1024683
Change-Id: I6ca506d74de463d914317f176eefbe74996298c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1879723
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
There are 3 layers modified, starting from the bottom up:
1. git_cache.py populate
Now takes a --no-fetch-tags option. If specified, the cache will not
fetch updated tags from the server by passing --no-tags to git fetch.
This prevents the server from sending all the tag refs. In chromium
this prevents significant time bottlenecks dealing with 10k+ tags.
2. bot_update.py
bot_update has to deal with multiple git repos, it has the root repo
that is checked out through git-cache, as well as repos synched via
DEPS and gclient.
The script now takes a --no_fetch_tags option. If specified,
the git-cache populate call will include --no-fetch-tags. Otherwise, it
won't. So this controls (for chromium) if fetches to the src.git server
are passed --no-tags.
3. bot_update/api.py
This is the entry point for recipes and also has to deal with multiple
git repos. The behaviour at this point is not changed from the default.
A |no_fetch_tags| parameter is added to ensure_checkout() that defaults
to False.
This CL is a refactor with no intended behavior change.
The next step will be to change the chromium build repo to pass True
for ensure_checkout(no_fetch_tags) on chromium trybots.
This represents solution #2 in
https://docs.google.com/document/d/1hDIunJjjfpmr50y3YnZ4o3aq1NZF4gJa1TS0p7AIL64/edit#
Bug: 1019824
Change-Id: I935430603299daa9e301a95a5184c0ce486fd912
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1894352
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: danakj <danakj@chromium.org>
Auto-Submit: danakj <danakj@chromium.org>
This reverts commit 1e4dbf3f64.
Reason for revert: unlikely but possible cause of http://shortn/_sgL4PqICVB
Original change's description:
> Add more git tracing.
>
> This CL adds more perf tracing to `git fetch` from bot_update.py and
> git_cache.py
>
> Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: infra
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
> Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Auto-Submit: Erik Chen <erikchen@chromium.org>
TBR=danakj@chromium.org,tandrii@google.com,erikchen@chromium.org
Change-Id: I03b33bf1532212f83a6b89b1de32fafb6b8aafc4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1891001
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
This CL adds more perf tracing to `git fetch` from bot_update.py and
git_cache.py
Change-Id: I9b69c60b6c81fc0c5f23f82fcc889b4d45a27556
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1888433
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Auto-Submit: Erik Chen <erikchen@chromium.org>
We currently fetch tags, which consumes ~17k lines. This limit ensures that we
won't blow up the log files in exceptional circumstances [currently setting
around ~2MB], but still allows us to get all the logs.
Change-Id: Ib690aaa07e2bde8549d221b90511b6c4863c3358
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: skia
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1867971
Commit-Queue: Erik Chen <erikchen@chromium.org>
Auto-Submit: Erik Chen <erikchen@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
This will allow us to separate the 'populate' and 'upload' phases
in the git cache updater recipe, which will allow us to do all the
'populate' steps in parallel, but then limit the parallelism of the
`pack/gc/upload` operations.
R=tandrii@chromium.org
Change-Id: I8b8a9155f86350be37ed5a67c592ff1fec4d42ff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1830857
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
I know that sometimes imports can have side-effects,
so unused imports shouldn't always be removed, but these
ones look like they could be.
Change-Id: Iea9f82afa99b0ea35f29a28f20ce0493b579cfee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1819860
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
This reverts commit 82ae4b4b7d.
Reason for revert: didn't help. Something else limits 'git gc' by 2GB.
However, even so, total amount of .pack files for chromium/src is <7.5GB.
Original change's description:
> git-cache: don't limit pack files to 2GiB.
>
> This is only relevant to 1 builder which updates git cache bundles.
> Since some time in 2018-2019, it started creating several 2GiB packs
> for chromium/src. I verified that the setting affecting it is
> core.deltaBaseCacheLimit
>
> Log file of a build which I observed while ssd-ed into the machine (internal):
> https://logs.chromium.org/logs/infra-internal/buildbucket/cr-buildbucket.appspot.com/8901682638839779824/+/steps/Updating_https:__chromium.googlesource.com_chromium_src/0/stdout
> Git version: version:2.21.0.chromium16
>
> R=ehmaldonado
>
> Change-Id: I52cadfb9f34faea09a57d53387cab7e0538362b9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1818076
> Auto-Submit: Andrii Shyshkalov <tandrii@google.com>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
TBR=tandrii@google.com,ehmaldonado@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I24915555c703bc92fda921524820405de5d69f7d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1817101
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This will be used on internal cache updater.
For instance, I've just compressed chromium/src resulting bootstrap
files from 20GiB to 7.5 GiB.
R=ehmaldonado
Change-Id: I15411700eb2ac3a26d1c658a12288cc49e48fd48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1802877
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Josip Sokcevic
Edward Lesmes
Edward Lesmes
Gavin Mak
Anthony Polito
Andrii Shyshkalov
Michael Moss
Andrii Shyshkalov
Robert Iannucci
John Budorick
danakj
Erik Chen
Quinten Yearsley
Karen Qian