Python multiprocessing is using sockets to communicate. UNIX socket names are
bound by PATH_MAX which includes temporary directory name.
gsutil uses multiprocessing, which creates file sockets in the tmpdir.
On some testing configurations, the tmpdir is set to a very long path, which
causes gsutil to fail with AF_UNIX path too long errors.
To prevent multiprocess failures, this CL overrides temporary directory
dictated by the environment to /tmp.
Bug: 812581
Change-Id: Idcd99d13b2e20b3095111fa26ec4e242848c8848
Reviewed-on: https://chromium-review.googlesource.com/923170
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Alexander Alekseev <alemate@chromium.org>
Although we want to prevent dfgs from untar'ing files to a parent
or sibling of its target directory, normal files that just happen
to have ".." in their name (i.e. not preceding a path separator) are
okay.
R=hinoka
Bug: 807286
Change-Id: Ibdc2c3615c4778ef66abceb532a4f671fbdab8ef
Reviewed-on: https://chromium-review.googlesource.com/912430
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
The first call to gsutil causes a network call and makes the download
script slow also in the most optimal cases. This CL refactors the
download script and moves the first gsutil call after checking locally
if sha1s match.
1) This turns the input acquisition into a generator and buffers the
files and sha1s in a list before multithreading.
2) This sequentially checks the sha1s and files and bails out early if
all match. In Chrome-land, we usually call this script with only one
file. There are some cases with around 4. This could also be
parallelized if the need arises.
3) The initial gsutil check, which ensures gsutil is updated, is moved
right in front of the multithreaded downloads.
The performance of one call to download_from_google_storage for an
existing 500MB file is 2.3s before this CL and 1.2s after (most of the
remaining time left is spent for making sha1sum).
Example for full gclient runhooks (when everything is up-to-date):
Chromium: 32s before, 12s after
V8: 12s before, 3s after
Bug: 776311
Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
Reviewed-on: https://chromium-review.googlesource.com/897562
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
This will allow all uses of this class to correctly move into
gsutil's personal vpython environment. In particular, the
get_toolchain_if_necessary script is pretty sneaky, and adds
depot_tools to sys.path to import and use this class. This
means that if get_toolchain_if_necessary is invoked in a vanilla
vpython environment (or an environment otherwise incompatible with
gsutil.py), it's use of this class will fail.
R=hinoka@chromium.org, tandrii@chromium.org
Bug: 795146
Change-Id: I87309a8b878b465cda7f53db0a41f2f9e6da6f50
Reviewed-on: https://chromium-review.googlesource.com/849663
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
`download_from_google_storage --config` is currently failing. It still uses gsutil 4.26 but 4.28 has been added recently:
https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/801355
Using 4.28 fixes the bug.
Bug: 796937
Change-Id: I299da7ede8087d832cd6e193d8fc4198a6878c6c
Reviewed-on: https://chromium-review.googlesource.com/840007
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Joao da Silva <joaodasilva@chromium.org>
Normally, if dfgs is started with multiple threads, the first
time gsutil.py is called, it may be called by many different
threads at the same time and hit a race condition where
multiple threads are all trying to download gsutil at the same
time. This works around the problem by calling it for the first
time in the main thread before anything else happens.
Bug: 773671
Change-Id: Ie75d90d6d20d11f46a601500b75a198fa5e5f111
Reviewed-on: https://chromium-review.googlesource.com/713750
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Apparently several scripts in other repros call this function directly.
Bug: 772741
Change-Id: I486483e44a072af6cf8c75373a2da6ef5469fc2c
TBR=dpranke
Reviewed-on: https://chromium-review.googlesource.com/707937
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
The script prints a bunch of stuff behind `if verbose` checks, but then
"verbose" was turned on by default in
https://bugs.chromium.org/p/chromium/issues/detail?id=504452
I think the wrong lesson was learned from that bug – it sounds like the
problem was that an error message was printed only if verbose was set.
After this change, the script is silent when it does nothing, and prints
something if something happens. (Arguably, it still prints too much in the
case where it successfully downloads something.)
This is part of a few changes to make `gclient runhooks` less noisy.
Bug: 772741,504452
Change-Id: I5823c296abe97611ba4411bab2743673b10dca4b
Reviewed-on: https://chromium-review.googlesource.com/706915
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
This picks up about 2 years worth of changes.
Bug:723733
TEST=Ran gclient sync on src.git checkout, worked.
Change-Id: I7021ac62be560bb3ba7b523f0758d56bceeef12a
Reviewed-on: https://chromium-review.googlesource.com/527262
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Without this change, the extracted directory could be missing but
the downloaded file could match the hash, and in turn
download_from_google_storage will not download the file and will
not extract it, and any tools depending on the extracted
directory will fail.
Review URL: https://codereview.chromium.org/1904483002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@300051 0039d316-1c4b-4281-b951-d872f2087c98
I fixed the overlapping argument and fixed up the tests so that the actual sha1 hash is now passed (since the tool actually check now that the downloaded file hash matches)
BUG=
R=hinoka@google.com
Review URL: https://codereview.chromium.org/1285423002 .
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296346 0039d316-1c4b-4281-b951-d872f2087c98
This does one last check to see if the file downloaded by
download_from_google_storage.py actually matches its sha1
BUG=
Review URL: https://codereview.chromium.org/1252313005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@296155 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
This conflicts with (and duplicates?) the -z option below and has broken one of the chrome android downstream bots.
Safer to revert than trying to fix
Original issue's description:
> Add support for tar.gz archive files to download from download_from_google_storage
>
> Also, add support for directly setting the public read flag on uploaded files.
>
> The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
>
> I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=295872
TBR=hinoka@google.com,hinoka@chromium.org,azarchs@chromium.org,ricow@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1209033006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295883 0039d316-1c4b-4281-b951-d872f2087c98
Also, add support for directly setting the public read flag on uploaded files.
The support for tar.gz allows the uploaded bundle to be a single tar.gz file instead of a lot of individual files using the directory support already in. The benefit here is that it is much easier to update the dependency. Simply clean out the existing files, copy in the new ones, create a tar.gz file, with the same name as the directory + 'tar.gz'. If the directory name and file name does not match up we will not clean up existing artifacts on download (i.e., there can be left over files after extracting).
I am doing this because I am moving a bunch of the dart dependencies to gcs, and a lot of our dependencies is much easier to manage with this in. If you don't like this, I can simply wrap the download script in another python script and do the logic there, but this may be handy for other people as well.
Review URL: https://codereview.chromium.org/807463005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295872 0039d316-1c4b-4281-b951-d872f2087c98
This does two noticable things:
* Prints a message when "download_from_google_storage --config" is run to
tell the user to enter "0" for the project ID prompt
* Removes the ".boto.depot_tools" boto file and defaults the boto file to
grant fullcontrol scopes.
Context: We restricted the depot_tools specific scopes to be readonly out of
concern that we would be forcing every developer to hold a set of non-expiring
write access credentials on their workstation. But this distinction has caused
a great deal of pain and anguish with confusing credentials (who would've thought
~/.boto.depot_tools would exist and might be broken?), and not for huge security
gains. Most people don't have write access to buckets, and the ones that do
definitely has a fullcontrol boto file already.
BUG=
Review URL: https://codereview.chromium.org/1182583003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295728 0039d316-1c4b-4281-b951-d872f2087c98
Currently check_bucket_permissions() in download_from_google_storage.py
performs a gsutil ls gs://bucket to determine whether the user has access
to the bucket or not.
This can be an EXTREMELY expensive operation (~minutes) if the bucket in
question has a lot of objects in the root (real case: chrome-telemetry).
It is worth noting that check_bucket_permissions() is not called just for
uploads but also for downloads, hence this is slowing down all invocations
of gclient sync on users and bots machine.
Removing the check_bucket_permissions() and let gsutil fail with an
Unauthorized error message if ACLs are not met.
BUG=458059
Review URL: https://codereview.chromium.org/923473002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294083 0039d316-1c4b-4281-b951-d872f2087c98
This prevents --no_auth from always clearing BOTO_CONFIG, since there
are times when a BOTO is needed for other things than just auth info
(e.g. proxy settings).
BUG=443523
R=hinoka@chromium.org, szager@chromium.org
Review URL: https://codereview.chromium.org/844373002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293652 0039d316-1c4b-4281-b951-d872f2087c98
The new GSUtil (or gs protocol, who knows) strips off the redundent x-goog-meta
string from the metadata key. This CL compensates for that.
Also since we're on 4.7, we can use the faster gsutil stat instead of gsutil ls -L.
BUG=
TEST=ran download_from_google_storage against compiler_proxy.sha, works
NOTREECHECKS=true
NOTRY=true
R=dnj@chromium.org, pgervais@chromium.org
Review URL: https://codereview.chromium.org/809123003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293432 0039d316-1c4b-4281-b951-d872f2087c98
This pins gsutil to a vanilla 4.7 instead of the weird custom 3.4 we have in depot_tools
BUG=
R=pgervais@chromium.org
Review URL: https://codereview.chromium.org/797663003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293413 0039d316-1c4b-4281-b951-d872f2087c98
--sha1_filename unnecessarily forces the output to have
the same filename and be in the same directory. The code
in main already correctly sets the file name to the
sha1_filename minus the .sha1, so the only change is
to actually use the --output path the same way the
rest of the code does.
R=iannucci@chromium.org
Review URL: https://codereview.chromium.org/752803003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293251 0039d316-1c4b-4281-b951-d872f2087c98
Checking bucket permissions takes ~400ms. We don't
need to do this if --no-auth because we know we
won't get a 403 and the 404 check will be handled
later when we try to actually download the file.
Also, remove the check for a null bucket. This
can't happen since we will throw a parser error
in the main function before we get to this code.
R=iannucci@chromium.org
Review URL: https://codereview.chromium.org/772203002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293250 0039d316-1c4b-4281-b951-d872f2087c98
A very common pattern for the users of this script is to have binaries in
three different folders, eg:
some/folder/win/bin.exe
some/folder/mac/bin
some/folder/linux/bin
Instead of using --platform to specify three different paths, we can recognize
this usage and pass in --auto_platform, and then --directory some/folder/ as the
target.
When enumerating files, it will match each individual file to see if they have
(win|mac|linux) as a parent folder name, and filter out non-matching platforms.
BUG=
TEST= src/third_party/clang_format/bin$ download_from_google_storage --auto_platform --no_auth -b chromium-clang-fomat -dr .
0> Downloading ./linux/clang-format...
Review URL: https://codereview.chromium.org/126893002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@245643 0039d316-1c4b-4281-b951-d872f2087c98
When download_from_google_storage --config is run, it should be implied that
the user just wants to download, not upload. This change passes the '-r' flag
into gsutil config, which requests a read-only scoped token rather than write
token. This is saved in ~/.boto.depot_tools so that it doesn't conflict
with a ~/.boto file crated later that may have write permissions.
BUG=
Review URL: https://codereview.chromium.org/135153002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@244276 0039d316-1c4b-4281-b951-d872f2087c98
Code path:
1. plugins.sso_auth is imported, which adds the AuthHandler class to the global state.
2. HasConfiguredCredentials() in gslib/utils.py is called by gsutil, and will return true if "prodaccess" exists on the system, which tells the system that we don't want a no-op auth handler.
3. When a command is called, all the auth handlers are cycled through and sso_auth.SSOAuth is called, which calls a stubby command to emit a gaiamint'ed oauth2 access token, which is then used as the Authorization Header
if --bypass_prodaccess is passed in, then:
1. HasConfiguredCredentials() will bypass the check for prodaccess, as if it didn't exist.
2. plugins.sso_auth does not get imported.
Which will essentially cause gsutil to behave as if this patch never existed.
So the expected behavior is:
=.boto file does not exist, prodaccess exists, but unauthenticated=
Failure: No handler was ready to authenticate. 3 handlers were checked. ['OAuth2Auth', 'HmacAuthV1Handler', 'SSOAuth'] Check your credentials.
=.boto file exists, prodaccess exists, but unauthenticated=
sso_auth will raise NotReadyToAuthenticate, and the .boto file will be used instead
=.boto file exists, prodaccess exists, authenticated=
sso_auth will be run _after_ the default gsutil authenticator, which causes the sso_auth to be used over whatever the default authentication is.
bypass_prodaccess is passed in by default to upload_to_google_storage because we expect people who use upload_to_google_storage to not need prodaccess and have their own boto file already. Also the sso_auth plugin will only request a readonlyi token, which will not work for uploading.
BUG=258152
Review URL: https://codereview.chromium.org/86123002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@240266 0039d316-1c4b-4281-b951-d872f2087c98
This should make Cygwin work to match "win32".
Review URL: https://codereview.chromium.org/108743002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@239244 0039d316-1c4b-4281-b951-d872f2087c98
This normalizes sys.platform so people running cygwin are treated like Windows.
Review URL: https://codereview.chromium.org/104763003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@238999 0039d316-1c4b-4281-b951-d872f2087c98
Adds a command line flag --platform to control which platforms the given download occurs on.
Automatically marks downloads executable on non-Windows if the header x-goog-meta-executable is set. Automatically set this flag on upload when the executable bit is set.
BUG=233100
Review URL: https://codereview.chromium.org/42273002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@231387 0039d316-1c4b-4281-b951-d872f2087c98
We appear to be getting 403's when encountering errors other than mismatched
credentials. The error message was originally hidden to avoid clutter, but it
has become obvious that there are other transient errors hidden behind a 403.
This change prints the original gsutil error message regardless of whether or not
we enconter a 403 or 404.
BUG=
Review URL: https://chromiumcodereview.appspot.com/23484061
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@224195 0039d316-1c4b-4281-b951-d872f2087c98
Able to pull internal tools, but distinct from the source that
those tools may eventually build.
TBR=cmp@chromium.org
BUG=252226
TEST=local buildslave
Review URL: https://chromiumcodereview.appspot.com/17351008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@209166 0039d316-1c4b-4281-b951-d872f2087c98
In order to support both developer workflow and not breaking bots, if the script
fails on a 403 in a bucket, it'll print a message asking developers to run
"download_from_google_storage --config" in order to create a new boto file.
This is not done automatically because it would break bots (Imagine hitting a 403,
and then gsutil wiping the .boto file, waiting for input, then dying).
BUG=231699,176331
Review URL: https://chromiumcodereview.appspot.com/17590010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@208806 0039d316-1c4b-4281-b951-d872f2087c98
It appears to be parsed and ignored, seems like a bug to me.
BUG=None
TEST=local/manual.
Review URL: https://chromiumcodereview.appspot.com/17265012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@207404 0039d316-1c4b-4281-b951-d872f2087c98
When a developer runs download_from_google_storage, and they don't have a .boto
file, the tool automatically runs "gsutil config" to create one for them.
Unfortunately, a side consequence is that if a bot runs the script, and it has
a boto file that 403's, then it would run "gsutil config" which moves the .boto
to .boto.bak, and creates an empty .boto file. This should not be the intended
action.
This CL changes so that "gsutil config" is not called, but instead just fails
with a message telling the dev to run that command.
TBR=maruel@chromium.org
Review URL: https://codereview.chromium.org/16072023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@203824 0039d316-1c4b-4281-b951-d872f2087c98
continuation of: https://codereview.chromium.org/11664024
Moved it from chrome/trunk/src/build to depot_tools/
BUG=153360
TEST=two unittests included in tests/
For end-to-end testing, check out a large directory. Run
find . -name .svn -prune -o -size +1000k -type f -print0 | upload_to_google_storage.py -b chrome-artifacts -0 -
(replacing chrome-artifacts with an upload-able bucket)
to test upload
run "find . -name .svn -prune -o -size +1000k -type f -print0 | xargs -0 rm" to remove the files uploaded. Check that the large binary files have been removed
run "download_from_google_storage.py -r -d -b chrome-artifacts ." to download the files again.
Review URL: https://chromiumcodereview.appspot.com/12042069
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@187951 0039d316-1c4b-4281-b951-d872f2087c98
Vasili Skurydzin
Alexander Alekseev
Aaron Gable
Michael Achenbach
Robert Iannucci
Joao da Silva
Ryan Tseng
Nico Weber
Rico Wind
pauljensen@chromium.org
dsansome@chromium.org
hinoka@chromium.org
ricow@chromium.org
boliu@chromium.org
nparker@chromium.org
primiano@chromium.org
mmoss@chromium.org
dpranke@chromium.org
ojan@chromium.org
hinoka@google.com
hinoka@chromium.org
brettw@chromium.org
scottmg@chromium.org
petermayo@chromium.org