This reverts commit f059ec9368.
Reason for revert: Reverting because we probably need to be using vpython and a compiled crcmod instead. See, e.g.,. b/188591640.
Original change's description:
> Update gsutil to use gsutil version 4.61, python3.
>
> This CL updates the gsutil.py wrapper to download and use
> v4.61 of GCP's gsutil, which is Python3-compatible.
>
> v4.61 appears to be fully self-contained and have all of the
> packages it needs vendored into it. So, there's no reason to
> use vpython anymore, and this CL removes that.
>
> Also, this CL removes the 'fallback' option to gsutil and
> the ability to force a version switch, as this should no
> longer be necessary (it was added for a migration back in 2014
> but apparently this code was never removed afterwards).
>
> This CL also updates download_from_google_storage.py and
> upload_to_google_storage.py to similarly not have the version flags
> and to just use regular python3, not vpython3.
>
> Bug: 1184108
> Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@google.com>
Bug: 1184108
Change-Id: I8e21a9a40d81e4e185642f866855b6838f80f1c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2905904
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Patch failures in bot_update return exit code 87 and 88, which are
ignored by ensure_checkout. This causes the bot_update step in recipes
to be successful despite a patch failure. This change removes these exit
codes from the accepted return codes for this step.
Bug: 1207685
Change-Id: I7bd71732a99184c96196659c6953e4d4b610c9b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893686
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This fixes CL:2880843 to use a module already present in vpython.
Bug: 1206782
Change-Id: If4158a62011e043e89f40f368c9c644434df6614
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2901758
Commit-Queue: Allen Webb <allenwebb@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
recipe_engine:
45b48dc4bf
45b48dc (hypan@google.com)
[cipd] Add `instances` support
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I17a52c2fe983436c7e6363a0cd2852687d09156f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2900986
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This CL updates the gsutil.py wrapper to download and use
v4.61 of GCP's gsutil, which is Python3-compatible.
v4.61 appears to be fully self-contained and have all of the
packages it needs vendored into it. So, there's no reason to
use vpython anymore, and this CL removes that.
Also, this CL removes the 'fallback' option to gsutil and
the ability to force a version switch, as this should no
longer be necessary (it was added for a migration back in 2014
but apparently this code was never removed afterwards).
This CL also updates download_from_google_storage.py and
upload_to_google_storage.py to similarly not have the version flags
and to just use regular python3, not vpython3.
Bug: 1184108
Change-Id: I0d1a8351dba2d3ad1f927afa333fb10959f19443
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2898439
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Enough works in that config that it seems like a better default by now.
Checking out, building, and uploading patches works (but uploading still
requires `--bypass-hooks` for now until crbug.com/1205263 is fixed).
(In case something important I'm not aware of breaks, we can revert this.)
Bug: 1103236,1205263
Change-Id: Id432ffd4ab19b15c019614953a2cefa787b46225
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2896468
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Nico Weber <thakis@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
recipe_engine:
beaf875152
beaf875 (iannucci@chromium.org)
[python3] Switch all uses of `six` to `future`.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ic916edf96367712b8f3d455fa495f6268df01ec2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2897999
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This change includes the following commits from the infra/infra
repository:
$ git log --format='%h %s (%ae)' \
97ee9cecaac4d99be09278169f987e61b8116672..HEAD \
-- go/src/infra/chromeperf/pinpoint/cli
a16d4a5458 Fix deadlock in Pinpoint CLI Unit Test (heiserya@google.com)
211e030fbe Add ability to run a batch of experiments to Pinpoint CLI (heiserya@google.com)
ca40b45ace pinpoint: Make artifact download concurrent (dberris@google.com)
08ea607681 pinpoint: print absolute path for downloaded files (fancl@chromium.org)
128352a9fb chromeperf: Add go_package option to protos (dberris@google.com)
6f69cd0a52 pinpoint: Initial support for downloading job artifacts (fancl@chromium.org)
fbcd876092 pinpoint: Compact job listing UI (dberris@google.com)
095bfe3bc8 pinpoint: Set a user agent for the CLI (dberris@google.com)
Change-Id: I4c31857272591667af5587ec83f3418cb1c14303
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2896109
Auto-Submit: Chenlin Fan <fancl@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Add revert instructions to .bat and bash files which are the entry
points to various infra tools deployed via CIPD.
Bug: 1206818
Change-Id: Ib0045aea86f29d0f7e3a43c6ff314c4b7aa74171
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2892387
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
os.path.realpath will resolve back to the subst paths, leading to
inappropriate current working directories for all downstream hooks.
BUG=1208688
Change-Id: Idfcb0fe4d25ab1317db5aae34663b1017d4c4937
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893669
Commit-Queue: Dirk Pranke <dpranke@google.com>
Auto-Submit: Robert Liao <robliao@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
I'm not saying this is why bot_update is failing, but I never trust python threads, especially not at scale. I also think the new implementation is easier to understand and less likely to have subtle mistakes if it is extended later
Change-Id: I29be55594a400fda845ea31893789104373fd0f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2892185
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
This makes the file easier to read.
Change-Id: Ie5eac66582bd6f3ce3c31def6f591e001de9a79a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893667
Auto-Submit: Robert Liao <robliao@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
The roll takes new feature to download Xcode & iOS runtimes separately.
The interface used by builder machines is untouched, but implementation
changed for new Xcode packages.
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 1191260
Change-Id: If0554e116bcb92c0b125693e1f555e52e22c10f4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2890331
Reviewed-by: Jeff Yoon <jeffyoon@google.com>
Commit-Queue: Zhaoyang Li <zhaoyangli@chromium.org>
If git push refs/for/ errors out, inspect stdout message to detect
uploadvalidator rejection. If present, present user a solution.
R=ajp@google.com
Change-Id: I70aee7615f3d905127a31bd5679968ade75c234a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2888475
Reviewed-by: Andy Perelson <ajp@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Gerrit rejects requests with 400 Bad request if query string `q`
contains any spaces. Replace spaces with a plus sign solves the problem.
R=thakis@chromium.org
Bug: 1199692
Change-Id: Ic13dda378527594c6cf57b8cb2edf740517811ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2832653
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Chromium + Fuchsia + Flutter Engine tools use GOMA_DIR to determine the
location of goma's scripts and binaries. Goma installs track some state
separately, and so when a different goma distribution's `compiler_proxy`
is launched and autoninja attempts to run `.cipd_bin/gomacc port`, the
check fails even though goma is running.
Additional context:
https://github.com/flutter/flutter/issues/81510#issuecomment-839489061
Change-Id: I93775628c6efba62bb226ac1995301c3ae7f5fa5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2891250
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Brandon DeRosier <bdero@google.com>
There were a couple of references to using filter() that
needed to be replaced with list comprehensions as the return
values may be used repeatedly.
Bug: 1207012
Change-Id: Iae0abd38aeb186d6f7945f8c848c1f7312cb45bb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2890329
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
recipe_engine:
955bddfc91~..dc36545f92232c37f4af4080296258fc1e240258
955bddf (olivernewman@google.com)
Do trivial s/master/main substitutions
2200d7d (bryner@google.com)
Upgrade cffi to 1.14.5 for the recipe engine.
dc36545 (iannucci@chromium.org)
[python3] Add test and do futurize --stage1.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org, tandrii@google.com
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: Icc1683726c3db84472f77333f6f6b9024f07dab3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2891350
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
Please review the expectation changes, and LGTM+CQ.
recipe_engine:
f4c9a77634
f4c9a77 (olivernewman@google.com)
[buildbucket] Make default git_ref consistent
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org, tandrii@google.com
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: I6b5526f445bbb7337ce8d18e10e855f7a91c587f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2885939
Auto-Submit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Commit-Queue: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
This is a followup to CL:2797935 to make it easier to debug exceptions.
Bug: 1206782
Change-Id: Ibb649e94fd562919737de152e3203ca6b02b38ad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2880843
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>
We now have an arm python binary, so this should no longer be needed.
Bug: 1103326,1184767,1103236
Change-Id: I5510bf2c61adc2d0d505a886956a6677303df271
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2888957
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
We have vpython wheels for py3 but not for py2 on mac/arm. With this,
download_from_google_storage.py works on arm mac even without the
vpython bypass. (Also, moving from py2 to py3 seems like a good thing
in general.)
Bug: 1205263,1103236
Change-Id: I6cb08b7c56e5e6fc160fb5abc86d68e04a01f8de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2881917
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This version will no longer enforce that wheels exist for linux-x86
when running in "verify" mode. This is a prerequisite to rolling out
new wheels to vpython specs that use verification, since we no longer
build for Linux 32-bit.
Bug: 1176014
Change-Id: I13e77fe7b9ccc3741fa05ae493d171d0ed9aecac
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2878769
Commit-Queue: Brian Ryner <bryner@google.com>
Auto-Submit: Brian Ryner <bryner@google.com>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
recipe_engine:
b64b15afdf
b64b15a (iannucci@chromium.org)
[python3] Add test and fix syntax+warnings for py3.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ibda2918e076d774608ec0b49f15beaefb7827b12
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2880846
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
This time, the mac/intel binary was built with an older macOS SDK
which makes the runtime believe that the macOS version on Big Sur is
10.16 instead of 11.0, which should work around the Big Sur issue we had
in the last attempt.
This version was pushed to bleeding_edge in
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2876311
Bug: 1124119,1103236,1184768,1175200
Change-Id: I8614d6f22704d46bfa78b81c8e4b9ff7e6668487
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2880559
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Otherwise, when the step completes, it won't be able to close
files used for stdout/stderr, since the metrics process is still
running.
Bug: 1206429
Change-Id: Id88578cbd9dab070e07f92e545ec137245566d16
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2877608
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (recipe_engine) into this repository.
recipe_engine:
972327732a
9723277 (iannucci@chromium.org)
[python3] Add a python3 unittest which does nothing.
More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.
R=iannucci@chromium.org
Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ic95165944ec7285f5e8ee37a36b3b92bc4598c8d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2879164
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
The new version is built with an older Xcode version for mac-amd64,
using the 10.15 SDK, which gives us better Big Sur compatibility for
some of the very old Python libraries we have in use. In particular,
it will see the OS version as 10.16 rather than 11.0, and recognize
the 10.x-built vpython wheels as compatible.
Bug: 1175200
Change-Id: I23be64b11c8a2fb31910ce5e13b98bd5b38eeb7e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2876311
Auto-Submit: Brian Ryner <bryner@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Edward Lesmes
Dirk Pranke
Alan Screen
Gavin Mak
Vadim Shtayura
Allen Webb
recipe-roller
Robert Iannucci
Erik Staab
Nico Weber
Evan Stade
Chenlin Fan
Brian Ryner
Nodir Turakulov
Robert Liao
Anthony Polito
Olivier Li
Zhaoyang Li
Josip Sokcevic
Brandon DeRosier
Mike Frysinger
Yoshisato Yanagisawa
Scott Lee