This reverts commit e8ef6259b9.
Reason for revert: https://crbug.com/1240673
High rate of bot_update failures, lots of gsutil defunct
processes.
Original change's description:
> Update gsutil to use gsutil version 4.66, python3
>
> This is a reland of 457736028d
> with following changes:
> * bump version from 4.61 to 4.66, which contains several bugfixes
>
> Original change's description:
> > Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
> >
> > This is a reland of e53a593956
> >
> > Additional bug fixes:
> > * handle race condition in gsutil when creating its directory
> > * limit to one gsutil process on darwin due to bug in python3
> >
> > Original change's description:
> > > Reland "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit af121aeec9.
> > >
> > > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> > >
> > > Original change's description:
> > > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > > >
> > > > This reverts commit f059ec9368.
> > > >
> > > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > > >
> > > > Original change's description:
> > > > > Update gsutil to use gsutil version 4.61, python3.
> > > > >
> > > > > This CL updates the gsutil.py wrapper to download and use
> > > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > > >
> > > > > v4.61 appears to be fully self-contained and have all of the
> > > > > packages it needs vendored into it. So, there's no reason to
> > > > > use vpython anymore, and this CL removes that.
> > > > >
> > > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > > the ability to force a version switch, as this should no
> > > > > longer be necessary (it was added for a migration back in 2014
> > > > > but apparently this code was never removed afterwards).
> > > > >
> > > > > This CL also updates download_from_google_storage.py and
> > > > > upload_to_google_storage.py to similarly not have the version flags
> > > > > and to just use regular python3, not vpython3.
> > > > >
> > > > > Bug: 1184108
> > > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> > Reviewed-by: Dirk Pranke <dpranke@google.com>
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Josip Sokcevic <sokcevic@google.com>
>
> Bug: 1184108
> Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108, 1240673
Change-Id: I74c3243ea29b99476e09b6ddb49cb052812a1e3e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3100348
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This is a reland of 457736028d
with following changes:
* bump version from 4.61 to 4.66, which contains several bugfixes
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Change-Id: I33787dc75f6e45d6b462706e934d7a2a37703fa7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3088085
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 457736028d.
Reason for revert: crbug/1227849 - android builders aren't happy,
also a report from user about android not working properly.
Original change's description:
> Reland "Reland "Update gsutil to use gsutil version 4.61, python3.""
>
> This is a reland of e53a593956
>
> Additional bug fixes:
> * handle race condition in gsutil when creating its directory
> * limit to one gsutil process on darwin due to bug in python3
>
> Original change's description:
> > Reland "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit af121aeec9.
> >
> > Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
> >
> > Original change's description:
> > > Revert "Update gsutil to use gsutil version 4.61, python3."
> > >
> > > This reverts commit f059ec9368.
> > >
> > > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> > >
> > > Original change's description:
> > > > Update gsutil to use gsutil version 4.61, python3.
> > > >
> > > > This CL updates the gsutil.py wrapper to download and use
> > > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > > >
> > > > v4.61 appears to be fully self-contained and have all of the
> > > > packages it needs vendored into it. So, there's no reason to
> > > > use vpython anymore, and this CL removes that.
> > > >
> > > > Also, this CL removes the 'fallback' option to gsutil and
> > > > the ability to force a version switch, as this should no
> > > > longer be necessary (it was added for a migration back in 2014
> > > > but apparently this code was never removed afterwards).
> > > >
> > > > This CL also updates download_from_google_storage.py and
> > > > upload_to_google_storage.py to similarly not have the version flags
> > > > and to just use regular python3, not vpython3.
> > > >
> > > > Bug: 1184108
> > > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> > >
> > > Bug: 1184108
> > > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Bug: 1184108
Bug: 1227849
Change-Id: Ib4176b94861e9f85caf6ed74050111a07b1e308d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3021641
Reviewed-by: Dirk Pranke <dpranke@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This is a reland of e53a593956
Additional bug fixes:
* handle race condition in gsutil when creating its directory
* limit to one gsutil process on darwin due to bug in python3
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ibb5d886fd22e3553521ff8ad6e2b4435844ef972
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988716
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit e53a593956.
Reason for revert: Speculative revert for build failures on Mac11 bots: crbug.com/1219065.
Original change's description:
> Reland "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit af121aeec9.
>
> Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
>
> Original change's description:
> > Revert "Update gsutil to use gsutil version 4.61, python3."
> >
> > This reverts commit f059ec9368.
> >
> > Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
> >
> > Original change's description:
> > > Update gsutil to use gsutil version 4.61, python3.
> > >
> > > This CL updates the gsutil.py wrapper to download and use
> > > v4.61 of GCP's gsutil, which is Python3-compatible.
> > >
> > > v4.61 appears to be fully self-contained and have all of the
> > > packages it needs vendored into it. So, there's no reason to
> > > use vpython anymore, and this CL removes that.
> > >
> > > Also, this CL removes the 'fallback' option to gsutil and
> > > the ability to force a version switch, as this should no
> > > longer be necessary (it was added for a migration back in 2014
> > > but apparently this code was never removed afterwards).
> > >
> > > This CL also updates download_from_google_storage.py and
> > > upload_to_google_storage.py to similarly not have the version flags
> > > and to just use regular python3, not vpython3.
> > >
> > > Bug: 1184108
> > > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > > Commit-Queue: Dirk Pranke <dpranke@google.com>
> >
> > Bug: 1184108
> > Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: Ib0757e0873d6e6b39621d50d327d959f0fa43720
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2957473
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit af121aeec9.
Reason for revert: re-landing with a switch back to vpython to get the compiled C extension version of crcmod for performance.
Original change's description:
> Revert "Update gsutil to use gsutil version 4.61, python3."
>
> This reverts commit f059ec9368.
>
> Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
>
> Original change's description:
> > Update gsutil to use gsutil version 4.61, python3.
> >
> > This CL updates the gsutil.py wrapper to download and use
> > v4.61 of GCP's gsutil, which is Python3-compatible.
> >
> > v4.61 appears to be fully self-contained and have all of the
> > packages it needs vendored into it. So, there's no reason to
> > use vpython anymore, and this CL removes that.
> >
> > Also, this CL removes the 'fallback' option to gsutil and
> > the ability to force a version switch, as this should no
> > longer be necessary (it was added for a migration back in 2014
> > but apparently this code was never removed afterwards).
> >
> > This CL also updates download_from_google_storage.py and
> > upload_to_google_storage.py to similarly not have the version flags
> > and to just use regular python3, not vpython3.
> >
> > Bug: 1184108
> > Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> > Reviewed-by: Mike Frysinger <vapier@chromium.org>
> > Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Commit-Queue: Dirk Pranke <dpranke@google.com>
>
> Bug: 1184108
> Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I5d6d6d06842e08517488471c144972818fcf02ff
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2907155
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit f059ec9368.
Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
Original change's description:
> Update gsutil to use gsutil version 4.61, python3.
>
> This CL updates the gsutil.py wrapper to download and use
> v4.61 of GCP's gsutil, which is Python3-compatible.
>
> v4.61 appears to be fully self-contained and have all of the
> packages it needs vendored into it. So, there's no reason to
> use vpython anymore, and this CL removes that.
>
> Also, this CL removes the 'fallback' option to gsutil and
> the ability to force a version switch, as this should no
> longer be necessary (it was added for a migration back in 2014
> but apparently this code was never removed afterwards).
>
> This CL also updates download_from_google_storage.py and
> upload_to_google_storage.py to similarly not have the version flags
> and to just use regular python3, not vpython3.
>
> Bug: 1184108
> Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This CL updates the gsutil.py wrapper to download and use
v4.61 of GCP's gsutil, which is Python3-compatible.
v4.61 appears to be fully self-contained and have all of the
packages it needs vendored into it. So, there's no reason to
use vpython anymore, and this CL removes that.
Also, this CL removes the 'fallback' option to gsutil and
the ability to force a version switch, as this should no
longer be necessary (it was added for a migration back in 2014
but apparently this code was never removed afterwards).
This CL also updates download_from_google_storage.py and
upload_to_google_storage.py to similarly not have the version flags
and to just use regular python3, not vpython3.
Bug: 1184108
Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
We have vpython wheels for py3 but not for py2 on mac/arm. With this,
download_from_google_storage.py works on arm mac even without the
vpython bypass. (Also, moving from py2 to py3 seems like a good thing
in general.)
Bug: 1205263,1103236
Change-Id: I6cb08b7c56e5e6fc160fb5abc86d68e04a01f8de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2881917
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
If download from google storage gets terminated while extracting .tar.gz
archive, it won't be extracted on the next run which results in invalid
workspace. This patches places a flag file if extraction wasn't
completed. Such flag is checked on next execution and forces re-download
if found.
This does not fix currently broken workspaces.
R=dpranke@google.com, ehmladonado@chromium.org
Bug: 1189273
Change-Id: I4047d98c193493d9a0fe20a46043e1f8376e5c7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774085
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
xrange doesn't exist in Python 3 and only nominally improves
memory usage on a value as small as MAX_TRIES.
Change-Id: I1e7408a05de880fee293466b1548c7f35e926270
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2246611
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This was made by running `codespell` and `scspell`
and then checking the results.
Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
The code in _downloader_worker_thread() does not work with Python 3:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/rakuco/src/depot_tools/download_from_google_storage.py", line 343, in _downloader_worker_thread
elif re.search(r'executable:\s*1', out):
File "/usr/lib/python3.7/re.py", line 183, in search
return _compile(pattern, flags).search(string)
TypeError: cannot use a string pattern on a bytes-like object
Bug: 984182, 1007872
Change-Id: I09df8169d802b010596ac4f34501b4bda805e6f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844833
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This throws an exception when running with Python 3:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/rakuco/src/depot_tools/download_from_google_storage.py", line 259, in _downloader_worker_thread
file_url = '%s/%s' % (base_url, input_sha1_sum.decode())
AttributeError: 'str' object has no attribute 'decode'
We are already calling bytes.decode() in enumerate_input(), whose output
is passed to _downloader_worker_thread().
Bug: 984182, 1007872
Change-Id: I55c4835192caf69a8dee9e89ae1bb5531f4bacae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1844832
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Without this we get an ugly stack trace. This is way more
friendly.
Bug: 983006
Change-Id: I5ba871ad71ad43bb48d91697e96473afbfe399a0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1697221
Auto-Submit: Eli Ribble <eliribble@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This enables gclient sync and gclient runhooks to run, barring hook script failures.
git cl upload also now works.
The scripts still work with Python 2.
There are no intended behaviour changes.
Bug: 942522
Change-Id: I2ac587b5f803ba7f5bb5e412337ce049f4b1a741
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524583
Commit-Queue: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Python multiprocessing is using sockets to communicate. UNIX socket names are
bound by PATH_MAX which includes temporary directory name.
gsutil uses multiprocessing, which creates file sockets in the tmpdir.
On some testing configurations, the tmpdir is set to a very long path, which
causes gsutil to fail with AF_UNIX path too long errors.
To prevent multiprocess failures, this CL overrides temporary directory
dictated by the environment to /tmp.
Bug: 812581
Change-Id: Idcd99d13b2e20b3095111fa26ec4e242848c8848
Reviewed-on: https://chromium-review.googlesource.com/923170
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Alexander Alekseev <alemate@chromium.org>
Although we want to prevent dfgs from untar'ing files to a parent
or sibling of its target directory, normal files that just happen
to have ".." in their name (i.e. not preceding a path separator) are
okay.
R=hinoka
Bug: 807286
Change-Id: Ibdc2c3615c4778ef66abceb532a4f671fbdab8ef
Reviewed-on: https://chromium-review.googlesource.com/912430
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
The first call to gsutil causes a network call and makes the download
script slow also in the most optimal cases. This CL refactors the
download script and moves the first gsutil call after checking locally
if sha1s match.
1) This turns the input acquisition into a generator and buffers the
files and sha1s in a list before multithreading.
2) This sequentially checks the sha1s and files and bails out early if
all match. In Chrome-land, we usually call this script with only one
file. There are some cases with around 4. This could also be
parallelized if the need arises.
3) The initial gsutil check, which ensures gsutil is updated, is moved
right in front of the multithreaded downloads.
The performance of one call to download_from_google_storage for an
existing 500MB file is 2.3s before this CL and 1.2s after (most of the
remaining time left is spent for making sha1sum).
Example for full gclient runhooks (when everything is up-to-date):
Chromium: 32s before, 12s after
V8: 12s before, 3s after
Bug: 776311
Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
Reviewed-on: https://chromium-review.googlesource.com/897562
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
This will allow all uses of this class to correctly move into
gsutil's personal vpython environment. In particular, the
get_toolchain_if_necessary script is pretty sneaky, and adds
depot_tools to sys.path to import and use this class. This
means that if get_toolchain_if_necessary is invoked in a vanilla
vpython environment (or an environment otherwise incompatible with
gsutil.py), it's use of this class will fail.
R=hinoka@chromium.org, tandrii@chromium.org
Bug: 795146
Change-Id: I87309a8b878b465cda7f53db0a41f2f9e6da6f50
Reviewed-on: https://chromium-review.googlesource.com/849663
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
`download_from_google_storage --config` is currently failing. It still uses gsutil 4.26 but 4.28 has been added recently:
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/801355
Using 4.28 fixes the bug.
Bug: 796937
Change-Id: I299da7ede8087d832cd6e193d8fc4198a6878c6c
Reviewed-on: https://chromium-review.googlesource.com/840007
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Joao da Silva <joaodasilva@chromium.org>
Normally, if dfgs is started with multiple threads, the first
time gsutil.py is called, it may be called by many different
threads at the same time and hit a race condition where
multiple threads are all trying to download gsutil at the same
time. This works around the problem by calling it for the first
time in the main thread before anything else happens.
Bug: 773671
Change-Id: Ie75d90d6d20d11f46a601500b75a198fa5e5f111
Reviewed-on: https://chromium-review.googlesource.com/713750
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Apparently several scripts in other repros call this function directly.
Bug: 772741
Change-Id: I486483e44a072af6cf8c75373a2da6ef5469fc2c
TBR=dpranke
Reviewed-on: https://chromium-review.googlesource.com/707937
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
The script prints a bunch of stuff behind `if verbose` checks, but then
"verbose" was turned on by default in
https://bugs.chromium.org/p/chromium/issues/detail?id=504452
I think the wrong lesson was learned from that bug – it sounds like the
problem was that an error message was printed only if verbose was set.
After this change, the script is silent when it does nothing, and prints
something if something happens. (Arguably, it still prints too much in the
case where it successfully downloads something.)
This is part of a few changes to make `gclient runhooks` less noisy.
Bug: 772741,504452
Change-Id: I5823c296abe97611ba4411bab2743673b10dca4b
Reviewed-on: https://chromium-review.googlesource.com/706915
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
This picks up about 2 years worth of changes.
Bug:723733
TEST=Ran gclient sync on src.git checkout, worked.
Change-Id: I7021ac62be560bb3ba7b523f0758d56bceeef12a
Reviewed-on: https://chromium-review.googlesource.com/527262
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Without this change, the extracted directory could be missing but
the downloaded file could match the hash, and in turn
download_from_google_storage will not download the file and will
not extract it, and any tools depending on the extracted
directory will fail.
Review URL: https://codereview.chromium.org/1904483002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@300051 0039d316-1c4b-4281-b951-d872f2087c98
I fixed the overlapping argument and fixed up the tests so that the actual sha1 hash is now passed (since the tool actually check now that the downloaded file hash matches)
BUG=
R=hinoka@google.com
Review URL: https://codereview.chromium.org/1285423002 .
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296346 0039d316-1c4b-4281-b951-d872f2087c98
This does one last check to see if the file downloaded by
download_from_google_storage.py actually matches its sha1
BUG=
Review URL: https://codereview.chromium.org/1252313005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296155 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
This conflicts with (and duplicates?) the -z option below and has broken one of the chrome android downstream bots.
Safer to revert than trying to fix
Original issue's description:
> Add support for tar.gz archive files to download from download_from_google_storage
>
> Also, add support for directly setting the public read flag on uploaded files.
>
> The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
>
> I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295872
TBR=hinoka@google.com,hinoka@chromium.org,azarchs@chromium.org,ricow@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1209033006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295883 0039d316-1c4b-4281-b951-d872f2087c98
Also, add support for directly setting the public read flag on uploaded files.
The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
Review URL: https://codereview.chromium.org/807463005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295872 0039d316-1c4b-4281-b951-d872f2087c98
This does two noticable things:
* Prints a message when "download_from_google_storage --config" is run to
tell the user to enter "0" for the project ID prompt
* Removes the ".boto.depot_tools" boto file and defaults the boto file to
grant fullcontrol scopes.
Context: We restricted the depot_tools specific scopes to be readonly out of
concern that we would be forcing every developer to hold a set of non-expiring
write access credentials on their workstation. But this distinction has caused
a great deal of pain and anguish with confusing credentials (who would've thought
~/.boto.depot_tools would exist and might be broken?), and not for huge security
gains. Most people don't have write access to buckets, and the ones that do
definitely has a fullcontrol boto file already.
BUG=
Review URL: https://codereview.chromium.org/1182583003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295728 0039d316-1c4b-4281-b951-d872f2087c98
Josip Sokcevic
Dirk Pranke
Nico Weber
Linkun Chen
Mirko Bonadei
Eli Ribble
Yuly Novikov
Quinten Yearsley
Takuto Ikuta
Sylvain Defresne
Edward Lesmes
Raphael Kubo da Costa
Edward Lemur
Victor Costan
Raul Tambre
Vasili Skurydzin
Alexander Alekseev
Aaron Gable
Michael Achenbach
Robert Iannucci
Joao da Silva
Ryan Tseng
Rico Wind
pauljensen@chromium.org
dsansome@chromium.org
hinoka@chromium.org
ricow@chromium.org
boliu@chromium.org