You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
Go to file
Ben Pastene 9519fc1300 Add timeouts to the actual http calls in gerrit_util.py
When gerrit's availability drops, chrome's builds see an excessive
amount of failures, eg:
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359780/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359594/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359564/overview
https://ci.chromium.org/ui/p/chromium/builders/try/linux-rel/1359630/overview

Seemingly all failures occur in either the `gerrit fetch current CL
info` step or the `gerrit changes` step. Both steps have a 60s timeout
and shell out to depot_tools' gerrit_client.py. That script essentially
makes a single http call to gerrit. That request has no configured
timeout. So when gerrit's MIA and the call hangs indefinitely, so too
will the step hang. 60s after that, the step timeout is reached, and the
entire build crashes with an infra-failure.

However, one single retry has been shown to sufficiently work around
at least one instance of that failure:
dea9a6eca26f690ce10a959701a8fb51ad/+/build.proto

So this incorporates timeouts into the requests made by
gerrit_util.py. Each request is given a 10s timeout, which should be
enough for most/all gerrit calls. (Both steps have a p90 of less than
1sec.) When a timeout is reached, the script will automatically retry
up to 4 times.

This also bumps the timeouts of the step calls to gerrit_client.py to
6min since the script can now take up to 5 minutes to fail, ie:
the sequence of sleeps is roughly 10s, 20s, 40s, 80s, 160s, which is
about 5min. So a 6min timeout should cover all those retries.

This also passes in "--verbose" to all step calls to this script, so
the logging that prints out the retry + sleep log lines will be
visible in build logs.

Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: chrome_release
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: infra
Bug: 1432638
Change-Id: I9dc47f4beeda3783ae4f9152bd29ee441ac3e197
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4420526
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
3 years ago
bootstrap Minor update to git. 3 years ago
fetch_configs Add gclient conversion script. 3 years ago
git-templates
infra Roll recipe dependencies (trivial). 3 years ago
man Revert "Add --no-squash argument to git rebase-update" 3 years ago
python-bin cipd/python: optimize cat usage 3 years ago
python2-bin cipd/python: optimize cat usage 3 years ago
recipes Add timeouts to the actual http calls in gerrit_util.py 3 years ago
testing_support Revert "Move scm.py to lib/" 3 years ago
tests Add timeouts to the actual http calls in gerrit_util.py 3 years ago
third_party [depot tools] Update six version. 3 years ago
win_toolchain Improve .boto message visibility 3 years ago
zsh-goodies
.cipd_impl.ps1 [cipd] Add windows-arm64 support to CIPD client bootstrap. 3 years ago
.gitattributes Add lib to recipe cipd package 3 years ago
.gitignore Redirect cros calls to py2.7 scripts to the vpython2.7 package. 3 years ago
.isort.cfg isort: make available to users 4 years ago
.style.yapf
.vpython Add explicit vpython verification tags which include mac-arm64. 4 years ago
.vpython3 Reland "Use py3 in gitiles recipe module" 4 years ago
CROS_OWNERS CROS_OWNERS: switch to our common shared lists 3 years ago
DIR_METADATA [depot_tools] Migrate metadata to DIR_METADATA. 5 years ago
GOMA_OWNERS Update GOMA_OWNERS 4 years ago
LICENSE
LUCI_OWNERS Updating LUCI_OWNERS 3 years ago
OWNERS Remove dpranke@ as a general-purpose OWNER. 3 years ago
PRESUBMIT.py Specify encoding when opening text files 3 years ago
README.gclient.md
README.git-cl.md Replace non-inclusive terms 5 years ago
README.md Indicate py3 compatibility for update_depot_tools_toggle.py 4 years ago
WATCHLISTS Update Source team WATCHLIST entry. 4 years ago
auth.py [auth] log errors from luci-auth token invocation. 5 years ago
autoninja [autoninja] Respect user-provided AUTONINJA_BUILD_ID 4 years ago
autoninja.bat Handle goma_ctl/autoninja /? 3 years ago
autoninja.py Create reclient specific ninja wrapper to properly handle reproxy lifecyle 3 years ago
bb Add revert instructions 5 years ago
bb.bat Add revert instructions 5 years ago
black black: add helper for formatting Python 4 years ago
bootstrap_python3 bootstrap_python3: Find 'cipd' in base_dir, not PATH 3 years ago
breakpad.py
cbuildbot
cipd Revert "Resolve symlinks to cipd, vpython3" 3 years ago
cipd.bat [cipd] Suppress findstr stdout. 3 years ago
cipd_bin_setup.bat Revert "Retry switching to use CIPD ninja v1.8.2" 3 years ago
cipd_bin_setup.sh
cipd_client_version [auth][oob-deprecation]: Update CIPD entries for LUCI Auth. 3 years ago
cipd_client_version.digests [auth][oob-deprecation]: Update CIPD entries for LUCI Auth. 3 years ago
cipd_manifest.txt [depot_tools] updating lucicfg version to v1.39.4 3 years ago
cipd_manifest.versions [depot_tools] updating lucicfg version to v1.39.4 3 years ago
cipd_manifest_cros_python2.txt Redirect cros calls to py2.7 scripts to the vpython2.7 package. 3 years ago
cipd_manifest_cros_python2.versions Redirect cros calls to py2.7 scripts to the vpython2.7 package. 3 years ago
cit Use py3 for remaining tools in depot_tools path 4 years ago
cit.bat Use py3 for remaining tools in depot_tools path 4 years ago
cit.py Use pylint 2.7 for depot_tools 4 years ago
clang-format Use py3 for remaining tools in depot_tools path 4 years ago
clang-format.bat Use py3 for remaining tools in depot_tools path 4 years ago
clang_format.py clang-format: lookup for binary under third_party/clang-format 3 years ago
clang_format_merge_driver Use py3 for remaining tools in depot_tools path 4 years ago
clang_format_merge_driver.bat Use py3 for remaining tools in depot_tools path 4 years ago
clang_format_merge_driver.py Use pylint 2.7 for depot_tools 4 years ago
codereview.settings update stale codereview.chromium.org links 5 years ago
compile_single_file Use py3 for remaining tools in depot_tools path 4 years ago
compile_single_file.bat Use py3 for remaining tools in depot_tools path 4 years ago
compile_single_file.py Call ninja.py from compile_single_file.py 3 years ago
cpplint.bat Use python3 for cpplint and get_toolchain on Win 4 years ago
cpplint.py Fix various lint false positives. 3 years ago
cpplint_chromium.py Fix various lint false positives. 3 years ago
cros Don't run cipd from depot_tools, it lives in a different location on Swarming. 3 years ago
cros_python2.vpython Redirect cros calls to py2.7 scripts to the vpython2.7 package. 3 years ago
cros_sdk
crosjobs Add revert instructions 5 years ago
detect_host_arch.py Use x64 emulation for Windows on Arm targets 3 years ago
dirmd Add revert instructions 5 years ago
dirmd.bat Add revert instructions 5 years ago
download_from_google_storage Switch download_from_google_storage to py3 5 years ago
download_from_google_storage.bat use python3 for download_from_google_storage.bat 4 years ago
download_from_google_storage.py Skip 'ls' when downloading from gs. 4 years ago
ensure_bootstrap Use python3 for gsutil bootstrapping 4 years ago
fetch
fetch.bat Reland "Make depot_tools scripts return exit codes properly on all platforms." 3 years ago
fetch.py fetch: make --no-history and --no-hooks switches consistent 3 years ago
fix_encoding.py Fix \r\r\n line endings in presubmit output 4 years ago
gclient Skip gclient auto-update for "getdep", "root", "help", "--help" 3 years ago
gclient-new-workdir.py Explicitly run everything with python3 4 years ago
gclient.bat Reland "Make depot_tools scripts return exit codes properly on all platforms." 3 years ago
gclient.py Revert "Use default number of cores for gclient jobs" 3 years ago
gclient_completion.sh
gclient_eval.py Fix roll-dep commit message on Windows and add suggestions 3 years ago
gclient_paths.py Suppress stderr of the git rev-parse command 3 years ago
gclient_scm.py Reset alternates when the git_cache dir is updated 3 years ago
gclient_utils.py Revert "Move scm.py to lib/" 3 years ago
gerrit Symlink gerrit CLI into depot_tools 3 years ago
gerrit_client.py Allow to pass CC list when create a gerrit change 4 years ago
gerrit_util.py Add timeouts to the actual http calls in gerrit_util.py 3 years ago
git-cache
git-cl
git-crrev-parse Fix git-crrev-parse to work with both master & main branches 4 years ago
git-drover
git-find-releases
git-footers
git-freeze
git-gs Add *.ts (typescript files) to git-gs 4 years ago
git-hyper-blame
git-map
git-map-branches
git-mark-merge-base
git-migrate-default-branch Add git-migrate-default-branch 5 years ago
git-nav-downstream
git-nav-upstream
git-new-branch
git-number
git-rebase-update
git-rename-branch
git-reparent-branch
git-retry
git-runhooks Revert "Remove unused functionality in bot_update.py" 4 years ago
git-squash-branch
git-thaw
git-upstream-diff
git_cache.py Reset git_cache mirror when HEAD points to master 3 years ago
git_cl.py Add newlines to info message so words aren't squashed together. 3 years ago
git_cl_completion.sh Add git completion helper to cl upload 6 years ago
git_common.py Don't enforce branch limit on rebase-update if only rebasing current 3 years ago
git_dates.py
git_drover.py Explicitly run everything with python3 4 years ago
git_find_releases.py Explicitly run everything with python3 4 years ago
git_footers.py Explicitly run everything with python3 4 years ago
git_freezer.py Explicitly run everything with python3 4 years ago
git_hyper_blame.py Explicitly run everything with python3 4 years ago
git_map.py Explicitly run everything with python3 4 years ago
git_map_branches.py git-map-branches: Fix --show-subject for branches that are gone 3 years ago
git_mark_merge_base.py Explicitly run everything with python3 4 years ago
git_migrate_default_branch.py Revert "Move scm.py to lib/" 3 years ago
git_nav_downstream.py Explicitly run everything with python3 4 years ago
git_new_branch.py Explicitly run everything with python3 4 years ago
git_number.py Use pylint 2.7 for depot_tools 4 years ago
git_rebase_update.py Revert "Add --no-squash argument to git rebase-update" 3 years ago
git_rename_branch.py Explicitly run everything with python3 4 years ago
git_reparent_branch.py Explicitly run everything with python3 4 years ago
git_retry.py Fix git-retry on errors 3 years ago
git_squash_branch.py Explicitly run everything with python3 4 years ago
git_upstream_diff.py Explicitly run everything with python3 4 years ago
gn gn: Call gn.bat in msysgit on Windows. 4 years ago
gn.bat Revert "Call python3.bat explicitly from gn.bat on Windows." 4 years ago
gn.py Use pylint 2.7 for depot_tools 4 years ago
goma_auth Reland "Reland "use python3 for goma related scripts"" 4 years ago
goma_auth.bat Set errorlevel in .bat wrappers on exit /b where appropriate. 4 years ago
goma_ctl Reland "Reland "use python3 for goma related scripts"" 4 years ago
goma_ctl.bat Handle goma_ctl/autoninja /? 3 years ago
gsutil.py Improve .boto message visibility 3 years ago
gsutil.py.bat Reland "Make depot_tools scripts return exit codes properly on all platforms." 3 years ago
gsutil.vpython3 Reland "Update gsutil to use gsutil version 4.68" 4 years ago
infra_to_superproject.py Print migration script steps. 3 years ago
isort isort: make available to users 4 years ago
led Add revert instructions 5 years ago
led.bat Add revert instructions 5 years ago
lockfile.py Reland "Use OS level locking in git_cache.py" 6 years ago
luci-auth Add revert instructions 5 years ago
luci-auth.bat Add revert instructions 5 years ago
lucicfg Add revert instructions 5 years ago
lucicfg.bat Add revert instructions 5 years ago
mac_toolchain
metrics.README.md Replace non-inclusive terms 5 years ago
metrics.py Update env_vars metrics name. 3 years ago
metrics_utils.py Replace metrics link to point to main 3 years ago
my_activity.py Use pylint 2.7 for depot_tools 4 years ago
ninja Revert "Use depot_tools python3 to call ninja.py" 3 years ago
ninja.bat Add single file compile support for Windows in ninja wrappers 3 years ago
ninja.py Fix a spelling error in a comment 3 years ago
ninja_reclient.py Use an md5 hash of the tmpdir for the autoninja reclient pipe on windows 3 years ago
ninjalog.README.md ninjalog_uploader: use goma_auth to detect googler 4 years ago
ninjalog_uploader.py [ninjalog_uploader] Exclude absolute paths and options from targets 3 years ago
ninjalog_uploader_wrapper.py Typo when countdown is more than 1. 4 years ago
owners_client.py Reland "Remove old DepotToolsOwners implementation" 3 years ago
owners_finder.py Use pylint 2.7 for depot_tools 4 years ago
pinpoint Add revert instructions 5 years ago
pinpoint.bat Add revert instructions 5 years ago
post_build_ninja_summary.py Add elapsed-time sorting option to build summary 3 years ago
presubmit_canned_checks.py Compress the request body if it's too large to send to luci-config 3 years ago
presubmit_support.py Revert "Extract searching relevat files in utils function" 3 years ago
profile.xml
prpc Add revert instructions 5 years ago
prpc.bat Add revert instructions 5 years ago
pylint-1.5 Fix hangs when running pylint with py3 4 years ago
pylint-1.5.bat Add an option to specify pylint version 4 years ago
pylint-2.6 Update pylint vpython specs to use a specific minor version. 3 years ago
pylint-2.6.bat Get pylint-2.6 to work under vpython3 on Windows. 4 years ago
pylint-2.7 Update pylint vpython specs to use a specific minor version. 3 years ago
pylint-2.7.bat Add pylint-2.7 4 years ago
pylint_main.py Explicitly run everything with python3 4 years ago
pylintrc CodeInclusion: Rename whitelist/blacklist -> allowlist/blocklist 5 years ago
python_runner.sh
rdb Add revert instructions 5 years ago
rdb.bat Add revert instructions 5 years ago
rdb_wrapper.py Add snippets to presubmit failures in resultdb 3 years ago
repo repo: restore fstrings for Python 3.6 5 years ago
repo_launcher Update repo_launcher to 2.32 3 years ago
roll-dep
roll-dep.bat
roll_dep.py Fix roll dep to use update-index 3 years ago
rustfmt.py Use rustfmt from third_party/rust-toolchain/bin directory. 3 years ago
scm.py Revert "Move scm.py to lib/" 3 years ago
setup_color.py Explicitly run everything with python3 4 years ago
split_cl.py Revert "Move scm.py to lib/" 3 years ago
subcommand.py Use empty argument list in cmdhelp 3 years ago
subprocess2.py [subprocess2] Replace VOID with DEVNULL 5 years ago
swift-format Support git cl formatting for swift files 4 years ago
swift_format.py Make swift-format default True on Mac hosts 4 years ago
update_depot_tools run `goma_ctl update_hook` in update_depot_tools 3 years ago
update_depot_tools.bat run `goma_ctl update_hook` in update_depot_tools 3 years ago
update_depot_tools_toggle.py Indicate py3 compatibility for update_depot_tools_toggle.py 4 years ago
upload_metrics.py Catch RemoteDisconnected on upload metrics 4 years ago
upload_to_google_storage.py Fix code check in upload to google storage. 3 years ago
utils.py Revert "Extract searching relevat files in utils function" 3 years ago
vpython Move python2_usage.txt from depot_tools to temp 3 years ago
vpython.bat Move python2_usage.txt from depot_tools to temp 3 years ago
vpython3 Revert "Resolve symlinks to cipd, vpython3" 3 years ago
vpython3.bat Use the bundled interpreter instead of passing -vpython-interpreter. 3 years ago
watchlists.py Explicitly run everything with python3 4 years ago
weekly Explicitly run everything with python3 4 years ago
whitespace.txt Change whitespace.txt 5 years ago
win32imports.py Reland "Use OS level locking in git_cache.py" 6 years ago
wtf Explicitly run everything with python3 4 years ago
yapf Update vpython spec for yapf to 0.31.0 3 years ago
yapf.bat Run yapf with vpython3 5 years ago

README.md

depot_tools

Tools for working with Chromium development. It requires python 2.7 or 3.8 for python 3 support.

Tools

The most important tools are:

  • fetch: A gclient wrapper to checkout a project. Use fetch --help for more details.
  • gclient: A meta-checkout tool. Think repo or git submodules, except that it support OS-specific rules, e.g. do not checkout Windows only dependencies when checking out for Android. Use gclient help for more details and README.gclient.md.
  • git cl: A code review tool to interact with Rietveld or Gerrit. Use git cl help for more details and README.git-cl.md.
  • roll-dep: A gclient dependency management tool to submit a dep roll, updating a dependency to a newer revision.

There are a lot of git utilities included.

Updating

depot_tools updates itself automatically when running gclient tool. To disable auto update, set the environment variable DEPOT_TOOLS_UPDATE=0 or run ./update_depot_tools_toggle.py --disable.

To update package manually, run update_depot_tools.bat on Windows, or ./update_depot_tools on Linux or Mac.

On Windows only, running gclient will install git and python.

Contributing

To contribute change for review:

git new-branch <somename>
# Hack
git add .
git commit -a -m "Fixes goat teleporting"
# find reviewers
git cl owners
git log -- <yourfiles>

# Request a review.
git cl upload -r reviewer1@chromium.org,reviewer2@chromium.org --send-mail

# Edit change description if needed.
git cl desc

# If change is approved, flag it to be committed.
git cl set-commit

# If change needs more work.
git rebase-update
...
git cl upload -t "Fixes goat teleporter destination to be Australia"

See also open bugs, open reviews, forum or report problems.

cpplint.py

Until 2018, our cpplint.py was a copy of the upstream version at https://github.com/google/styleguide/tree/gh-pages/cpplint. Unfortunately, that repository is not maintained any more. If you want to update cpplint.py in depot_tools, just upload a patch to do so. We will figure out a long-term strategy via issue https://crbug.com/916550.

Note that the cpplint.py here is also used by the Tricium analyzer, so if the cpplint.py here changes, we should also update the copy used there.