aiden
/
depot_tools
mirror of https://chromium.googlesource.com/chromium/tools/depot_tools
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
main
chrome/4147
chrome/3987
chrome/3865
chrome/3904
infra/config
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
depot_tools/README.google_cloud_storage.md

3.6 KiB
Raw Permalink Blame History

Google Cloud Storage usage in Chromium

This doc explains the components that help Chromium developer's and Chromium infrastructure interact with Google Cloud Storage.

When will I encounter Google Cloud Storage when developing Chromium

There may be different hooks run by gclient that pull necessary artifacts from Google Cloud Storage. gclient runhooks or gclient sync command would download the files, depending on your .gclient config.

In other cases the Chromium CI build recipes also read and write objects to Google Cloud Storage.

Components

These components should have the same effect whether they're run on a Swarming bot or a developer's Chromium checkout. These components do not affect the usage of the standard gsutil or gcloud storage binaries outside of Chromium workflows. In those cases, you would need to use gsutil config or gcloud auth login directly.

gsutil.py

gsutil.py wraps an unmodified version of gsutil which handles authentication support with LUCI Auth and pins the gsutil version.

All gsutil arguments, except for config, are passed through to the actual gsutil binary. The config argument will redirect to a standard luci-auth login interactive flow and generate a token with https://www.googleapis.com/auth/devstorage.full_control scope. This scope is necessary for the wide range of commands gsutil supports.

Every command sent through to the gsutil binary is wrapped with luci-auth context. Context starts a local LUCI auth context server which creates a temporary BOTO file which in turn fools gsutil into fetching its access tokens from the context server. When the command is finished the BOTO file is deleted and the local context server stopped so there's nothing to persist and risk leaking. Note: The context server is always running on bots.

You can view this happening here:

root@8447532ea598:~/chromium$ inotifywait -m -r /tmp/ | grep .boto
Setting up watches.
Watches established.
/tmp/luci56666486/gsutil-luci-auth/ OPEN .boto
/tmp/luci56666486/gsutil-luci-auth/ ACCESS .boto
/tmp/luci56666486/gsutil-luci-auth/ CLOSE_NOWRITE,CLOSE .boto
/tmp/luci56666486/gsutil-luci-auth/ OPEN .boto
/tmp/luci56666486/gsutil-luci-auth/ ACCESS .boto
/tmp/luci56666486/gsutil-luci-auth/ CLOSE_NOWRITE,CLOSE .boto
/tmp/luci56666486/gsutil-luci-auth/ DELETE .boto

The code that actually performs this can be found below in the LUCI auth gsutil integration section.

Note: gsutil.py can use any .boto file provided in the home directory or via the BOTO_CONFIG environment variable. You should not need either of these in most cases, furthermore they will likely cause problems for you so we strongly suggest removing them if already set.

download_from_google_storage.py

download_from_google_storage.py is a convenience wrapper library around gsutil.py. If you are not working in Python then it's probably best to use the gsutil.py command-line interface (CLI) described above.

LUCI Auth gsutil Integration

luci-auth gsutil integration library is a shim library that's used to enable a 3-legged OAuth flow using LUCI auth refresh tokens rather than the standard gcloud auth login refresh tokens or gsutil config generated BOTO file.