Commit Graph

175 Commits (951578030eaea669cde44a8eb79aaff69bfbdfeb)

Author SHA1 Message Date
Robert Iannucci ad3ad021d9 [gerrit_util] Invert Authenticator model.
Instead of having CreateHttpConn manipulate the request, with the
Authenticator only able to provide the Authorization header value,
the Authenticator now gets the ability to manipulate the entire
HttpConn object.

This will be used for a new Authenticator method which needs to
include a proxy, cookies, and also manipulate the target request
URI, in addition to providing an Authorization header value.

R=ayatane@chromium.org, yiwzhang@google.com

Bug: 336351842
Change-Id: Ia7d0bbfbb907d8ab6c6d12d000f514fa7afc7245
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5585665
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Allen Li <ayatane@chromium.org>
11 months ago
Robert Iannucci 840e538154 [gerrit_util] Add stronger type annotations around CreateHttpConn.
This also fixes a potential bug where ReadHttpJsonResponse could
improperly return None if the server had a completely empty reply.
ReadHttpJsonResponse will now return an empty dictionary in this
case (which is the assumption that most of the callsites are
making).

R=yiwzhang@google.com

Bug: 336351842
Change-Id: I0aa88e233563a0685b6c0f32ea77ad3e094b9cbc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5585184
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
11 months ago
Gregory NISBET 3963cc6399 [gerrit_util.py] Remove unnecessary imports from gerrit_util.py
Remove the libraries that have no uses according to gd in vim.

BUG=None
TEST=upload this CL

Change-Id: I7fe6bbd54c1a4f00a3d0e3f04be0f02609734e46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5578897
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gregory Nisbet <gregorynisbet@google.com>
11 months ago
Robert Iannucci 6427b94bc2 [git_cl] Refactor EnsureAuthenticated.
This will allow us to invert the main Authenticator API.

R=yiwzhang@google.com

Bug: 336351842
Change-Id: Iebe06f68373b99685809849bcd96371e7634d6ef
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5582406
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
11 months ago
Robert Iannucci c57b7ed364 [gerrit_util] Change Authenticator API to return proxy info.
This will be used with an upcoming SSOAuthenticator implementation
which will need to proxy all http requests for Googlers.

R=ayatane, gavinmak@google.com

Bug: 336351842
Change-Id: If8cbb8db51fce198e704f109232868421130b40c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5582100
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
11 months ago
Robert Iannucci a85dcffff4 [git_cl] Refactor away a use of isinstance.
Previously when composing a debugging trace, it would use an
isinstance check to special-case inclusion of a gitcookies file.

This CL refactors this to be part of the Authenticator API, instead,
which means that we will always get some sort of authenticator log
file in a trace, even if it's empty.

This also provides an affordance to add debugging information for
the other authenticator types later.

R=ddoman@chromium.org, gavinmak@google.com

Bug: 336351842
Change-Id: Idd6f45ea60b089f9b2391b5527c5281f67421043
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5571497
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
11 months ago
Robert Iannucci 3ea849f363 [git_cl.py] Remove all usage and messages around netrc.
netrc was already legacy in 2017:
fe30f18fc5

The removed gerrit-init.sh script was written in 2013 and no longer
works.

R=yiwzhang@google.com

Bug: 336351842
Change-Id: Ia8e845899973eb5ead66aee2f54078c09d80ef28
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5564197
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
11 months ago
Fei Shao b4fbce4794 gerrit_util: Expand user path from http.cookiefile config
This fixes an issue where "~" in user http.cookiefile config doesn't get
expanded and causes failures when "git cl" commands tries to read the
credentials.

Expand the path returned by scm.GIT.GetConfig() but not just the default
value.

Bug: b/341619336
Change-Id: Ib89d5b86710a0a6bb6ca669639eeec6adc75689d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5547345
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Fei Shao <fshao@chromium.org>
11 months ago
Josip Sokcevic ab914fad1e Improve cloudtop error message on git cl failure
R=gavinmak@google.com

Fixed: 280621923
Change-Id: I856dc1eb4f3fcd65906196b66f358b1dd413012e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5547690
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
12 months ago
Josh Berenhaus 03ee6d950d Fix use of deprecated LOGGER.warn in gerrit test
Change-Id: Ibe60a15f80ffb6072094c53c7b80fcc9419fc52f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5380997
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Josh Berenhaus <joshber@meta.com>
1 year ago
Aravind Vasudevan 9f9bab25da Reland "Update gclient to use git config caching"
This reverts commit 3569608028.

Reason for revert: This includes a fix for crbug.com/324358728.

The rebase-update command has logic which tries to specifically set a key to an empty string and this has been intentionally set this way[1]. The new SetConfig implementation does treats empty string as None and hence tries to unset the config, resulting in error code 5. The patchset 2 fixes this bug and adds a test to ensure SetConfig can set an empty string to be backward compatible.

[1] https://codereview.chromium.org/228353003

Original change's description:
> Revert "Update gclient to use git config caching"
>
> This reverts commit 3edda8d185.
>
> Reason for revert: Breaks rebase-update; crbug.com/324358728
>
> Original change's description:
> > Update gclient to use git config caching
> >
> > This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
> >
> > This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
> >
> > A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
> >
> > Benchmarking
> > ============
> > With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
> >
> > Windows Benchmark
> > =================
> > Baseline (gpaste/6360045736951808): ~1min 12 sec.
> > With Caching (gpaste/6480065209040896): ~1min 3sec.
> > ~12.5% decrease in gclient sync noop runtime.
> >
> > Linux Benchmark
> > ===============
> > Baseline (gpaste/4730436763254784): ~3.739 sec.
> > With Caching (gpaste/4849870978940928): ~3.534 sec.
> > ~5.5% decrease in gclient sync noop runtime.
> >
> > Bug: 1501984
> > Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> > Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: 1501984
> Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1501984
Change-Id: I405abc16c2ef6f0689031c82c61af71aad302122
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280779
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Aravind Vasudevan 3569608028 Revert "Update gclient to use git config caching"
This reverts commit 3edda8d185.

Reason for revert: Breaks rebase-update; crbug.com/324358728

Original change's description:
> Update gclient to use git config caching
>
> This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.
>
> This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.
>
> A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.
>
> Benchmarking
> ============
> With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.
>
> Windows Benchmark
> =================
> Baseline (gpaste/6360045736951808): ~1min 12 sec.
> With Caching (gpaste/6480065209040896): ~1min 3sec.
> ~12.5% decrease in gclient sync noop runtime.
>
> Linux Benchmark
> ===============
> Baseline (gpaste/4730436763254784): ~3.739 sec.
> With Caching (gpaste/4849870978940928): ~3.534 sec.
> ~5.5% decrease in gclient sync noop runtime.
>
> Bug: 1501984
> Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Bug: 1501984
Change-Id: I4a603238d9ed43edafc8e574493800670520a1d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5279198
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Aravind Vasudevan 3edda8d185 Update gclient to use git config caching
This change updates all the modules used by gclient to use `scm.GIT` for git config calls over directly invoking the subprocess.

This change currently doesn't modify git_cache since the config reads and writes within it are done on bare repository. A follow-up CL will update git_cache.

A follow-up CL will also update git_cl and git_map_branches since they have shown performance improvements too: https://crrev.com/c/4697786.

Benchmarking
============
With chromium/src as the baseline super project, this change reduces about 380 git config calls out of 507 total calls on cache hits during no-op. The below numbers are benchmarked with `update_depot_tools` turned off.

Windows Benchmark
=================
Baseline (gpaste/6360045736951808): ~1min 12 sec.
With Caching (gpaste/6480065209040896): ~1min 3sec.
~12.5% decrease in gclient sync noop runtime.

Linux Benchmark
===============
Baseline (gpaste/4730436763254784): ~3.739 sec.
With Caching (gpaste/4849870978940928): ~3.534 sec.
~5.5% decrease in gclient sync noop runtime.

Bug: 1501984
Change-Id: Ib48df2d26a0c742a9b555a1e2ed6366221c7db17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5252498
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Josip Sokcevic 0cba8dd68e [owners] Limit number of concurrent Gerrit connections
R=gavinmak@google.com

Fixed: 319469729
Change-Id: I955fa17c80cee25bc2163a8bca0234cdbb659f62
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5186760
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak 7f5b53ff8d Remove last py2 workarounds
python3 is the only supported version of python in depot_tools.

Changes include:
  * updating python version requirements
  * removing third_party/six

Bug: 1475402
Change-Id: Ib86231413fcac494ceb27d1783e53b9d86533c26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4845292
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak cc97655889 Reland "Drop py2 support in gerrit and git related files"
This is a reland of commit b5c7f4b46c

Original change's description:
> Drop py2 support in gerrit and git related files
>
> python3 is the only supported version of python in depot_tools.
>
> Bug: 1475402
> Change-Id: Ie4ee18d297081b3aa0206b8d7ce6461819bff0ca
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809560
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Bug: 1475402
Change-Id: I194180494071777b7b9dd91a5c8edabbbf5484c1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4811218
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Gavin Mak 42674f5d2d Revert "Drop py2 support in gerrit and git related files"
This reverts commit b5c7f4b46c.

Reason for revert: missing a replace for urlparse.urlparse

Original change's description:
> Drop py2 support in gerrit and git related files
>
> python3 is the only supported version of python in depot_tools.
>
> Bug: 1475402
> Change-Id: Ie4ee18d297081b3aa0206b8d7ce6461819bff0ca
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809560
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Bug: 1475402
Change-Id: Idd00fdfe0b3d62785da2789a7dfcc9fbc79b6385
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4811623
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Gavin Mak b5c7f4b46c Drop py2 support in gerrit and git related files
python3 is the only supported version of python in depot_tools.

Bug: 1475402
Change-Id: Ie4ee18d297081b3aa0206b8d7ce6461819bff0ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809560
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Gavin Mak fc75af35d4 Bump Gerrit fetch retry logic to 6 tries and 12s initial sleep
Upon Gerrit team's suggestion, up the retry time and number of retries
to handle increased replication delay.

Bug:b/285164390
Change-Id: If833ae9bb0f8c276b761970a8b5f96ec217d11b9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4621268
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Gavin Mak <gavinmak@google.com>
2 years ago
Xinan Lin dbcecc9017 Allow 409 error from Gerrit when creating the branch
Gerrit may return 409 error to us even if the branch was created.
We should ignore this error if the branch head is the commit we
requested.

Also pre-check the branch status before creating it.
If an existing branch is expected by the caller, simply return
the branch info without sending the creation request.

BUG=1433047
TEST=local run
Change-Id: Ic6184705579f1410cb9c54843cc0bbb4c4ad61b1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4436317
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Garrett Beaty <gbeaty@google.com>
Commit-Queue: Xinan Lin <linxinan@chromium.org>
2 years ago
Xinan Lin 1344a3c998 Add global 300s global timeout for Gerrit API calls
This will not impact the calls which have explicit timeout.
For the rest, either PUT or POST or DELETE methods, 5 min should
suffice.

Fixed: 1441863
TEST=NA

Change-Id: I235bc60344b6981eb3d96930ca8b6463129a2be2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4495994
Commit-Queue: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Ben Pastene 97dadd025f Apply the gerrit REST timeout to only 'GetChanges' calls
https://crrev.com/c/4420526 put a 10s timeout on all calls made via
this script.

But it appears there are a variety of endpoints that take more than
10s, the latest being "SubmitChange", eg:
https://ci.chromium.org/ui/p/chromeos/builders/release/release-R113-15393.B-orchestrator/25/overview

So instead of applying the timeout to everything, and then opting
certain long-running calls out of it, this just opts-in the
"GetChanges" call to the timeout. This should cover the failure mode
that was originally solved for chrome's bots. And also bump the
timeout to 30s since we don't trust all get-changes queries to
reliably finish in under 10s.

This also bumps the step level timeout for the query in tryserver
recipe_mod to 8 min since there could be ~5min worth of sleeping +
150s worth of waiting+timing-out. So worst case the step will now
take 8min to fully exhaust all timeouts/sleeps.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: b/278083716
Change-Id: Ib366e004e0bb07297ba732590d488cae779e38ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4426524
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
2 years ago
Ben Pastene 04182559c5 Don't put a timeout on 'CreateChange' gerrit rest calls
https://crrev.com/c/4420526 put a 10s timeout on all calls made via
this script. But the "CreateChange" endpoint can sometimes take 1+ min.

eg: One of the calls in the following build took 1.1 min:
https://ci.chromium.org/ui/p/chrome/builders/official.infra/chrome-release/8203/overview

So this removes the timeout for CreateChange calls since it's
unclear what a good timeout for that would be.

Bug: 1433047
Change-Id: If6e68663a11add1d72691e97018dba73cfed4f21
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4423859
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
2 years ago
Ben Pastene 9519fc1300 Add timeouts to the actual http calls in gerrit_util.py
When gerrit's availability drops, chrome's builds see an excessive
amount of failures, eg:
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359780/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359594/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359564/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359630/overview

Seemingly all failures occur in either the `gerrit fetch current CL
info` step or the `gerrit changes` step. Both steps have a 60s timeout
and shell out to depot_tools' gerrit_client.py. That script essentially
makes a single http call to gerrit. That request has no configured
timeout. So when gerrit's MIA and the call hangs indefinitely, so too
will the step hang. 60s after that, the step timeout is reached, and the
entire build crashes with an infra-failure.

However, one single retry has been shown to sufficiently work around
at least one instance of that failure:
dea9a6eca26f690ce10a959701a8fb51ad/+/build.proto

So this incorporates timeouts into the requests made by
gerrit_util.py. Each request is given a 10s timeout, which should be
enough for most/all gerrit calls. (Both steps have a p90 of less than
1sec.) When a timeout is reached, the script will automatically retry
up to 4 times.

This also bumps the timeouts of the step calls to gerrit_client.py to
6min since the script can now take up to 5 minutes to fail, ie:
the sequence of sleeps is roughly 10s, 20s, 40s, 80s, 160s, which is
about 5min. So a 6min timeout should cover all those retries.

This also passes in "--verbose" to all step calls to this script, so
the logging that prints out the retry + sleep log lines will be
visible in build logs.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: 1432638
Change-Id: I9dc47f4beeda3783ae4f9152bd29ee441ac3e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4420526
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
2 years ago
Aravind Vasudevan a02b4bf030 Fix gerrit new-password URL
This change fixes gerrit new-password URL from <host>-review.googlesource.com/new-password to <host>.googlesource.com/new-password.

Fixed: 1412557
Change-Id: I0c73e890fa9db5e2172d9a88cbc50703675e9f50
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4219808
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
2 years ago
Gavin Mak e0fee9fa4b Add BatchListBestOwners and ListBestOwners to GerritClient
ListOwners returns all OWNERS for a file, which is undesirable in cases
when only direct OWNERS are needed.

Bug: 1351212, 1351519
Change-Id: I693b6645c780aa589e8ab24d0b58691f4aeb30f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3823299
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Leszek Swirski c1c45f8a85 [gerrit] Add more gerrit API wrappers
Add helpers to gerrit_util for:

  * Setting the commit message on a ChangeEdit,
  * Creating a cherry-pick,
  * Getting the contents of a file in a change.

Bug: v8:12849
Change-Id: I1a5de3054a366d480def32942fd42be179ffb749
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683383
Auto-Submit: Leszek Swirski <leszeks@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Leszek Swirski 4c0c3fbb06 [gerrit] Fix encoding in ChangeEdit
gerrit_util.ChangeEdit was passing the result of b64encode into a %s
format, which was printing the contents with a bytes prefix (b'...').

Decode the bytes object back to a str object before passing it to '%s'

Bug: v8:12849
Change-Id: I8b54b427bbbe8b914444d7486b5a8afbd743de70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683382
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Leszek Swirski <leszeks@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
3 years ago
Aravind Vasudevan c5f0cbb865 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 42c5bbbc96 Revert "Use pylint 2.7 for depot_tools"
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan 22bf605bb6 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Michael Moss 9c28af45e6 Add ability to generically call most Gerrit APIs.
This will make it easier to call new or uncommon Gerrit APIs, without
needing to add dedicated methods for each one. This also makes it easier
to debug API calls by exposing the raw response data, which is often
processed or filtered in the existing wrapper methods.

BUG=1052179
R=linxinan@chromium.org

Change-Id: I0b594a9fdaabae4b0ebb379dc5473f4d6de67a13
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3235714
Auto-Submit: Michael Moss <mmoss@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Xinan Lin <linxinan@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
4 years ago
Michael Moss b6ce244503 Add 'create tag' support to gerrit_client and gerrit recipe_module.
BUG=1052179
R=dpranke@google.com, linxinan@chromium.org

Change-Id: I6e220b70e63d990e9257d9fe4d107de7dbd2eb81
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3232849
Auto-Submit: Michael Moss <mmoss@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
4 years ago
Xinan Lin 2b4ec95534 Add new subcommand for 'Changes Submitted Together'.
We may need to submit a chain of CLs. With this sub-cmd, we could
track their revisions along the chain, without querying them one
by one.

BUG=1236658
TEST=local run

Change-Id: If5930fb8f01daccff4d7c5c5e91c8db8887d8e47
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3098095
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Xinan Lin <linxinan@chromium.org>
4 years ago
Xinan Lin af79f24cae GetGerritBranch should return None if the branch does not exist
Currently gerrit_client retries if received 404, which does not
provide anything meaningful.
It is nature to get 404 for a non-exist branch. I am not sure why
we raise a 200 error per an empty response. I think
we could just accept 404 and return it like other GET functions, e.g.
GetAccountDetails().

BUG=1208430
TEST=local

Change-Id: I054bad99b69c54cc125141108299193f5cc092de
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3077363
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Xinan Lin <linxinan@chromium.org>
4 years ago
Xinan Lin 1bd4ffa295 Remove wait-for-merge from gerrit client tool
Following comment in crrev.com/c/3039331, submit call
is fully synchronous.

BUG=1207955
TEST=e2e

Change-Id: I9146d43b4d4530ad5064709e418d34e3229e19ef
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3049672
Commit-Queue: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
4 years ago
Xinan Lin 0b0738d9fe Add setbotcommit method to gerrit_client
This is required if we want to land a bot-generated-cl.

BUG=1207955
TEST=e2e

Change-Id: Ib758b9782f9951a3ec23ce9970b0db2a48b1b5f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3049573
Commit-Queue: Xinan Lin <linxinan@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Michael Moss <mmoss@chromium.org>
4 years ago
Xinan Lin c2fb26a337 gerrit_client could retrieve the branch and tag info for a commit hash
BUG=1207955
TEST=verified at local

Change-Id: Ife31456371f156d342ddd099bba1521051d700fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3044919
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Xinan Lin <linxinan@chromium.org>
4 years ago
Raphael Kubo da Costa 9f6aa1ba78 gerrit_util: Make GceAuthenticator also consider proxy errors in httplib2
Some HTTP proxies may return an HTTP error such as 503 when DNS
resolution fails for GceAuthenticator._INFO_URL
(http://metadata.google.internal), which causes e.g. git-cl to throw an
exception rather than falling back to checking for .netrc or
.gitcookies.

Make GceAuthenticator._get() also consider an httplib2 proxy error a
sign that the code is not running in GCE.

Change-Id: Icc094829f9b044176be44e51f3cad49d4a374f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2982611
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Marco Georgaklis 85557a08f4 Implemented suppport for getrelatedchanges endpoint in gerrit module
The endpoint documentation can be found here:
https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#get-related-changes

This involved adding support for the command in gerrit_client and
gerrit_util and then incorporating functionality and testing into
the gerrit recipe module.

The reasoning for this is to be able to identify chained builds for
improved accuracy of the new test identification for the FLake
Endorser project.

Bug:1204163
Change-Id: Ic60c733540a9afd7ff159408b0ea9ad7aac078e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2928016
Commit-Queue: Marco Georgaklis <mgeorgaklis@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
4 years ago
Josip Sokcevic f5c6d8a27d Handle spaces in Gerrit search options
Gerrit rejects requests with 400 Bad request if query string `q`
contains any spaces. Replace spaces with a plus sign solves the problem.

R=thakis@chromium.org

Bug: 1199692
Change-Id: Ic13dda378527594c6cf57b8cb2edf740517811ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2832653
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
LaMont Jones 9eed4238d8 gerrit: add createchange changeedit publishchangeedit
Add calls to support creating changes, as well as editing and publishing
them in gerrit_client and gerrit_util.

Bug=b:182613582

Change-Id: I0514cf08dce63ab29d99d4485d96fa124006326a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2800811
Auto-Submit: LaMont Jones <lamontjones@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: LaMont Jones <lamontjones@chromium.org>
4 years ago
Edward Lesmes 743e98ce87 gerrit_util: Fix checking if code-owners enabled on repo.
Change-Id: I5f518c22be4c5496f91202015c9caf18d0fa1be9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2778638
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Edward Lesmes 8170c29fd1 Reland "presubmit: Skip owners checks if code-owners plugin is enabled."
This is a reland of 2cf835a9ba

URL-encoded repo path, so chromium/src becomes chromium%2Fsrc
as Gerrit expects

Original change's description:
> presubmit: Skip owners checks if code-owners plugin is enabled.
>
> If code-owners plugin is enable for the repo, skip owners check on
> commit, and skip checking owners format, as that will be done by
> the plugin.
>
> Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I3038590f3a92cbf7b6dc0ba6eb47f72593a2ccf7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775840
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
4 years ago
Edward Lesmes 23c3bdc950 owners: Simplify logic to score and suggest owners.
Keep adding owners in the order suggested by Gerrit until coverage is
achieved instead of finding a minimal set of owners, as that takes too
long

Bug: 1186420
Change-Id: Id2ab172db0b2e83ab950191958813163099181be
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2751343
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Josip Sokcevic ec3c39536a Revert "presubmit: Skip owners checks if code-owners plugin is enabled."
This reverts commit 2cf835a9ba.

Reason for revert: doesn't work with depot_tools

error 404 Not found: tools (with applied patch from 2729404)

Bug: 1183975

Original change's description:
> presubmit: Skip owners checks if code-owners plugin is enabled.
>
> If code-owners plugin is enable for the repo, skip owners check on
> commit, and skip checking owners format, as that will be done by
> the plugin.
>
> Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: Id4d560701e4b0144e0aafc5263e09ed6927f6222
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2729409
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
4 years ago
Edward Lesmes 2cf835a9ba presubmit: Skip owners checks if code-owners plugin is enabled.
If code-owners plugin is enable for the repo, skip owners check on
commit, and skip checking owners format, as that will be done by
the plugin.

Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Gavin Mak 7d690053cb owners_client: Handle missing email in AccountInfo
Bug: 1180316
Change-Id: I8390f72d8a6cd1044ce10b7cbd4573a004a9730c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2705614
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
4 years ago