If gsutil is not downloaded, and if gsutil (or
download_from_google_storage) is called concurrently with n>2, it's
possible that those processes are doing the same work. There is also a
critical point where gsutil.py can fail with:
Destination path '/depot_tools_path/external_bin/gsutil/gsutil_4.68/d'
already exists error.
To avoid this problem, use FS locking around code that manipulates with
files.
R=jojwang@google.com
Bug: 338040708
Change-Id: Ib83aaa1e09628f878e512d79f2fa5221c2bcfd37
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5502531
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
On unsupported platforms, luci-auth is not downloaded from CIPD and
hence the scripts relying on gsutil.py fail. Given we aren't using
luci-auth in unsupported platforms anyway, skip wrapping gsutil with
luci-auth on unsupported platforms.
Curently, only "aix" is added to the unsupported platforms list
eventhough we have other unsupported platforms too. This list will be
expanded if necessary.
R=yiwzhang
Change-Id: I70f6fceddc672c76bbea9a108a75b84662e26459
Fixed: 1467752
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4989412
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py
Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.
Reformatted using:
files=( $(
git ls-tree -r --name-only HEAD | \
grep -Ev -e '^(third_party|recipes)/' | \
grep '\.py$';
git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"
The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py
These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py
Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
python3 is the only supported version of python in depot_tools.
Bug: 1475402
Change-Id: I71db631b5556525dd4932134679c70cacd205a90
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4824616
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This reverts commit dfafd0a3c7.
Reason for revert: gsutil.py was changed to be non-executable, this is breaking all of our builds which rely on gsutil.py. Example: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64%20flutter_packaging_test
If making it non-executable was intended, feel free to close this.
Original change's description:
> [gsutil] Update gsutil error message on outdated boto
>
> Add information where .boto file is located
>
> Change-Id: Id8605196ab0906a594bbb0c87a016986415eedac
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4787527
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Change-Id: I88e0a993a67a85e3c81799f6f3f128c4f9276ff7
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4791811
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
An out of date .boto file can prevent Google employees from being able
to download the Chromium toolchain from Google storage. An error message
was added to gsutil.py to help developers diagnose this, but the message
was swallowed by get_toolchain_if_necessary.py. This change prints the
error message, and adds '*' * 80 separators to make it more visible.
An obsolete import is also removed.
Bug: 1414152
Change-Id: I69d5a786672304ed363c291e0a8ef89f8be0f2e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4310059
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This change cleans up GSUTIL_ENABLE_LUCI_AUTH flag from gsutil. This is the final change to transition to luci-auth.
Fixed: 1359383
Change-Id: I1b4ac4bf95a887ebec92f71df343e855829846a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4197458
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
gsutil.py adds a line break at the end of stdout/stderr when luci-auth is used. This change fixes it.
R=sokcevic, jojwang
Bug: 1412249, 1359383
Change-Id: Ieb200db94d9ed9149e17df93c33802a7433a8d5d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4221262
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
This change removes OOB deprecation log from gsutil. This is no longer necessary and it hinders with tools that use gsutil.py.
R=sokcevic, jojwang
Bug: 1412249, 1359383
Change-Id: If47239520a15d24658314bd44570babf99831e36
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4221926
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This reland includes fix for crbug.com/1410703. It fallbacks to normal gsutil call when luci-auth is not logged in.
This is a reland of commit a7b20b34f8
Original change's description:
> Enable luci-auth by default
>
> This change enables luci-auth by default for gsutil.py, replacing OOB authentication flow.
>
> Bug: 1359383
> Change-Id: I1a93edc0d04e03c13921359205f252b9e1ed7594
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4193195
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Bug: 1359383
Change-Id: I9acfa8a578de8e106346b83aae509f9ffe1146e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4199427
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This is a reland of commit 5e5802d7a2
This was reverted as a temporary fix for servers running python 3.5. This reland removes `encoding` arg from `subprocess.run` temporarily until all servers are migrated to a supported python version.
Original change's description:
> Update log when luci-auth is not logged-in
>
> This change updates gsutil.py to capture when luci-auth is not logged in and prints useful log for signing in. This change couple enables luci-auth UI when luci-auth is enabled.
>
> Bug: 959170, 1359383
> Change-Id: Ic636bb362df388073e1fd11d736a47f35b749016
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4154550
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 959170, 1359383
Change-Id: I409a49362f0c584bd39f1031671e9e089cf50f97
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4165881
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
This reverts commit 5e5802d7a2.
Reason for revert: This CL introduces a dependency on Python version 3.10. I'm not clear that this is allowed for depot_tools. We are using it on systems with Python 3.5.
Original change's description:
> Update log when luci-auth is not logged-in
>
> This change updates gsutil.py to capture when luci-auth is not logged in and prints useful log for signing in. This change couple enables luci-auth UI when luci-auth is enabled.
>
> Bug: 959170, 1359383
> Change-Id: Ic636bb362df388073e1fd11d736a47f35b749016
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4154550
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 959170, 1359383
Change-Id: I0a73051f6b65786374f2b585f13ca88f5f8f83d5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4165720
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This change updates gsutil.py to capture when luci-auth is not logged in and prints useful log for signing in. This change couple enables luci-auth UI when luci-auth is enabled.
Bug: 959170, 1359383
Change-Id: Ic636bb362df388073e1fd11d736a47f35b749016
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4154550
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
The change updates gsutil.py to internally use luci-auth for
authentication. It redirects `gsutil.py config` to `luci-auth login` and
wraps gsutil.py calls using `luci-auth context`.
The change is mainly done to ensure users have a way to authenticate once OOB authentication flow is deprecated. This patch falls back to using the .boto file when that is provided.
Design Document: go/gsutil-oauth-issue-resolved
Bug: 959170, 1359383
Change-Id: Icc6df1d8b93424ac0f6e2df84afdebc869763ede
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4109192
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
This is a reland of commit 8edc960e9a
Original change's description:
> Update gsutil to use gsutil version 4.68
>
> We can't yet upgrade to version 5.x since it's only py3 compatbile and
> some builders still rely on running gsutil with py2.
>
> R=gavinmak@google.com
>
> Bug: 1184108
> Change-Id: I002cc3625a2184ab42383b88a014c0631d1a33ee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3508516
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Change-Id: I8f4b726c203ed12d415deeec249972fe855b59c8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3514940
Reviewed-by: Brian Ryner <bryner@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 8edc960e9a.
Reason for revert: potentially breaking chromeos builders
Original change's description:
> Update gsutil to use gsutil version 4.68
>
> We can't yet upgrade to version 5.x since it's only py3 compatbile and
> some builders still rely on running gsutil with py2.
>
> R=gavinmak@google.com
>
> Bug: 1184108
> Change-Id: I002cc3625a2184ab42383b88a014c0631d1a33ee
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3508516
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Change-Id: I40ce645ebd928c63116012353c632026c787b154
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3514508
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
We can't yet upgrade to version 5.x since it's only py3 compatbile and
some builders still rely on running gsutil with py2.
R=gavinmak@google.com
Bug: 1184108
Change-Id: I002cc3625a2184ab42383b88a014c0631d1a33ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3508516
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit e8ef6259b9.
Reason for revert: https://crbug.com/1240673
High rate of bot_update failures, lots of gsutil defunct
processes.
Original change's description:
> Update gsutil to use gsutil version 4.66, python3
>
> This is a reland of 457736028d
> with following changes:
> * bump version from 4.61 to 4.66, which contains several bugfixes
>
> Original change's description:
> > Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
> >
> > This is a reland of e53a593956
> >
> > Additional bug fixes:
> > * handle race condition in gsutil when creating its directory
> > * limit to one gsutil process on darwin due to bug in python3
> >
> > Original change's description:
> > > Reland "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit af121aeec9.
> > >
> > > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> > >
> > > Original change's description:
> > > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > > >
> > > > This reverts commit f059ec9368.
> > > >
> > > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > > >
> > > > Original change's description:
> > > > > Update gsutil to use gsutil version 4.61, python3.
> > > > >
> > > > > This CL updates the gsutil.py wrapper to download and use
> > > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > > >
> > > > > v4.61 appears to be fully self-contained and have all of the
> > > > > packages it needs vendored into it. So, there's no reason to
> > > > > use vpython anymore, and this CL removes that.
> > > > >
> > > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > > the ability to force a version switch, as this should no
> > > > > longer be necessary (it was added for a migration back in 2014
> > > > > but apparently this code was never removed afterwards).
> > > > >
> > > > > This CL also updates download_from_google_storage.py and
> > > > > upload_to_google_storage.py to similarly not have the version flags
> > > > > and to just use regular python3, not vpython3.
> > > > >
> > > > > Bug: 1184108
> > > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> > Reviewed-by: Dirk Pranke <dpranke@google.com>
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Josip Sokcevic <sokcevic@google.com>
>
> Bug: 1184108
> Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108, 1240673
Change-Id: I74c3243ea29b99476e09b6ddb49cb052812a1e3e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3100348
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This is a reland of 457736028d
with following changes:
* bump version from 4.61 to 4.66, which contains several bugfixes
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 457736028d.
Reason for revert: crbug/1227849 - android builders aren't happy,
also a report from user about android not working properly.
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Bug: 1227849
Change-Id: Ib4176b94861e9f85caf6ed74050111a07b1e308d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3021641
Reviewed-by: Dirk Pranke <dpranke@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This is a reland of e53a593956
Additional bug fixes:
* handle race condition in gsutil when creating its directory
* limit to one gsutil process on darwin due to bug in python3
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit e53a593956.
Reason for revert: Speculative revert for build failures on Mac11 bots: crbug.com/1219065.
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ib0757e0873d6e6b39621d50d327d959f0fa43720
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2957473
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit af121aeec9.
Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
Original change's description:
> Revert "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit f059ec9368.
>
> Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
>
> Original change's description:
> > Update gsutil to use gsutil version 4.61, python3.
> >
> > This CL updates the gsutil.py wrapper to download and use
> > v4.61 of GCP's gsutil, which is Python3-compatible.
> >
> > v4.61 appears to be fully self-contained and have all of the
> > packages it needs vendored into it. So, there's no reason to
> > use vpython anymore, and this CL removes that.
> >
> > Also, this CL removes the 'fallback' option to gsutil and
> > the ability to force a version switch, as this should no
> > longer be necessary (it was added for a migration back in 2014
> > but apparently this code was never removed afterwards).
> >
> > This CL also updates download_from_google_storage.py and
> > upload_to_google_storage.py to similarly not have the version flags
> > and to just use regular python3, not vpython3.
> >
> > Bug: 1184108
> > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit f059ec9368.
Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
Original change's description:
> Update gsutil to use gsutil version 4.61, python3.
>
> This CL updates the gsutil.py wrapper to download and use
> v4.61 of GCP's gsutil, which is Python3-compatible.
>
> v4.61 appears to be fully self-contained and have all of the
> packages it needs vendored into it. So, there's no reason to
> use vpython anymore, and this CL removes that.
>
> Also, this CL removes the 'fallback' option to gsutil and
> the ability to force a version switch, as this should no
> longer be necessary (it was added for a migration back in 2014
> but apparently this code was never removed afterwards).
>
> This CL also updates download_from_google_storage.py and
> upload_to_google_storage.py to similarly not have the version flags
> and to just use regular python3, not vpython3.
>
> Bug: 1184108
> Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This CL updates the gsutil.py wrapper to download and use
v4.61 of GCP's gsutil, which is Python3-compatible.
v4.61 appears to be fully self-contained and have all of the
packages it needs vendored into it. So, there's no reason to
use vpython anymore, and this CL removes that.
Also, this CL removes the 'fallback' option to gsutil and
the ability to force a version switch, as this should no
longer be necessary (it was added for a migration back in 2014
but apparently this code was never removed afterwards).
This CL also updates download_from_google_storage.py and
upload_to_google_storage.py to similarly not have the version flags
and to just use regular python3, not vpython3.
Bug: 1184108
Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This enables gclient sync and gclient runhooks to run, barring hook script failures.
git cl upload also now works.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: I2ac587b5f803ba7f5bb5e412337ce049f4b1a741
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524583
Commit-Queue: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Picks up about 2 years worth of changes. Should be relatively risk-free since
we pin gsutil.py in almost all of our scripts.
Bug:
Change-Id: Iff1365ee6d6f31e03da8d8876b6ccd0b47a033c2
Reviewed-on: https://chromium-review.googlesource.com/801355
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>
Josip Sokcevic
Gaby Baghdadi
Aravind Vasudevan
Gavin Mak
Mike Frysinger
Drew Roen
Bruce Dawson
William Hesse
Joanna Wang
Josip Sokcevic
Garrett Beaty
Takuto Ikuta
Dirk Pranke
Edward Lemur
Edward Lesmes
Quinten Yearsley
Raul Tambre
Caleb Rouleau
Ryan Tseng