You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Ben Pastene 9519fc1300 Add timeouts to the actual http calls in gerrit_util.py ... When gerrit's availability drops, chrome's builds see an excessive amount of failures, eg: https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359780/overview https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359594/overview https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359564/overview https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359630/overview Seemingly all failures occur in either the `gerrit fetch current CL info` step or the `gerrit changes` step. Both steps have a 60s timeout and shell out to depot_tools' gerrit_client.py. That script essentially makes a single http call to gerrit. That request has no configured timeout. So when gerrit's MIA and the call hangs indefinitely, so too will the step hang. 60s after that, the step timeout is reached, and the entire build crashes with an infra-failure. However, one single retry has been shown to sufficiently work around at least one instance of that failure: dea9a6eca26f690ce10a959701a8fb51ad/+/build.proto So this incorporates timeouts into the requests made by gerrit_util.py. Each request is given a 10s timeout, which should be enough for most/all gerrit calls. (Both steps have a p90 of less than 1sec.) When a timeout is reached, the script will automatically retry up to 4 times. This also bumps the timeouts of the step calls to gerrit_client.py to 6min since the script can now take up to 5 minutes to fail, ie: the sequence of sleeps is roughly 10s, 20s, 40s, 80s, 160s, which is about 5min. So a 6min timeout should cover all those retries. This also passes in "--verbose" to all step calls to this script, so the logging that prints out the retry + sleep log lines will be visible in build logs. Recipe-Nontrivial-Roll: build Recipe-Nontrivial-Roll: build_limited Recipe-Nontrivial-Roll: chrome_release Recipe-Nontrivial-Roll: chromiumos Recipe-Nontrivial-Roll: infra Bug: 1432638 Change-Id: I9dc47f4beeda3783ae4f9152bd29ee441ac3e197 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4420526 Reviewed-by: Joanna Wang <jojwang@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org>		2 years ago
..
full.expected	Add timeouts to the actual http calls in gerrit_util.py	2 years ago
full.py	Update recipes python tests.	2 years ago