In https://crbug.com/1327371, we started encountering hanging gsutil
calls. These were triggered under very specific circumstances, but
were essentially due to a multi-processing bug in python on MacOS:
https://bugs.python.org/issue33725
When running gsutil on mac, it explicitly suggests disabling
multi-processing for this reason:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/gsutil/gslib/command.py;drc=3a12d6ccdec28da8bda09d9ff826aae1f9504e59;l=1331
So this CL simply puts that suggestion into practice. There are two
options for doing so: adding a line to an active Boto file, or adding
a "-o" arg to every gsutil invocation. Since chromium has multiple
copies of gsutil and multiple different locations where `gsutil` is
invoked, the latter option is considered intractable.
Instead, we choose the Boto file option by adding a context manager
to the gsutil recipe module that will insert a custom Boto file into
the environment that only disables multi-threading. Due to the fact
that the BOTO_PATH env var takes multiple paths, we can safely
incorporate both our custom Boto along with LUCI's per-build Boto file
in the same gsutil invocation. And when the context manager exits, the
environment should revert back to its original state.
Chromium incorporates this method in crrev.com/c/3662023. See it take
effect during the "gclient runhooks" step in this led build:
https://ci.chromium.org/swarming/task/5b0c5174e70cb010?server=chromium-swarm.appspot.com
Bug: 1327371
Change-Id: I75cfdf16d0ec023bf81ea57991dde996694a46c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3661949
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
The storage.* links do not work unless an individual object is
specified. This isn't the case for recursive upload/copy operations.
When uploading multiple objects recursively, we should use the console
link instead.
Bug: None
Test: Updated recipe expectations.
Recipe-Nontrivial-Roll: build
Change-Id: I511f04f367f6c20c538b3c2093a40807323d7b75
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2118421
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Alex Drexler <adrexler@google.com>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
In support of the signing rework we're going to want to have
the signers keep track of the calculated hash to ensure that
we don't have a TOCTOU type issue. This commit adds the stat
command to the gsutil module so that we can ask gs a files hash.
Change-Id: Ib35d1541292a18fbfc26fab50f594c682ba4109d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1754262
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: George Engelbrecht <engeg@google.com>
This caused a bit of confusion for a customer who thought that they were
forbidden from passing Path objects in cmd.
R=tandrii@chromium.org
Change-Id: I352a6d5be75f48dcc0c8a8c92d6921359585ed48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1536493
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
I'm writing a recipe that makes heavy use of gsutil's ls and cat, and
some of the calls are expected to fail. So this allows the user to make
those that fail do so as normal steps.
R=iannucci, tandrii
Bug: 866062
Change-Id: I4c9cdcb77c00afd4ab8fefd146f22fa94fcc37d7
Reviewed-on: https://chromium-review.googlesource.com/c/1325332
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
This will help remove confusion when depot_tools' recipes.py
ends up in PATH.
BUG=699120
Change-Id: Id4c21b0cc6bb022ea2c21145abe76bebb0a8d9c1
Reviewed-on: https://chromium-review.googlesource.com/458430
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Ben Pastene
Aravind Vasudevan
Gavin Mak
Quinten Yearsley
Alex Drexler
Marc-Antoine Ruel
George Engelbrecht
Robert Iannucci
Michael Achenbach
Paweł Hajdan, Jr