You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Achenbach 1f067b88df Optimize check for existing files in download_from_google_storage ... The first call to gsutil causes a network call and makes the download script slow also in the most optimal cases. This CL refactors the download script and moves the first gsutil call after checking locally if sha1s match. 1) This turns the input acquisition into a generator and buffers the files and sha1s in a list before multithreading. 2) This sequentially checks the sha1s and files and bails out early if all match. In Chrome-land, we usually call this script with only one file. There are some cases with around 4. This could also be parallelized if the need arises. 3) The initial gsutil check, which ensures gsutil is updated, is moved right in front of the multithreaded downloads. The performance of one call to download_from_google_storage for an existing 500MB file is 2.3s before this CL and 1.2s after (most of the remaining time left is spent for making sha1sum). Example for full gclient runhooks (when everything is up-to-date): Chromium: 32s before, 12s after V8: 12s before, 3s after Bug: 776311 Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317 Reviewed-on: https://chromium-review.googlesource.com/897562 Commit-Queue: Michael Achenbach <machenbach@chromium.org> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Reviewed-by: Ryan Tseng <hinoka@chromium.org>		7 years ago
..
gstools	Scripts to download files from google storage based on sha1 sums	12 years ago
auth_test.py	[depot_tools] Fix timestamp issues related to local auth	8 years ago
bot_update_coverage_test.py	Reland 2 "bot_update recipe: Upload source manifest"	8 years ago
checkout_test.py	Revert "Fix checkout.py issues when p.patchlevel > 1."	8 years ago
download_from_google_storage_unittest.py	Optimize check for existing files in download_from_google_storage	7 years ago
fix_encoding_test.py	Standardize the sys.path fix up and fix a few pylint warnings.	14 years ago
gclient_eval_unittest.py	gclient conditionals are supposed to allow '!=', but don't currently.	8 years ago
gclient_scm_test.py	Fix pylint error in gclient_scm_test.	7 years ago
gclient_smoketest.py	Allow custom_vars to override vars in flattened output.	7 years ago
gclient_test.py	Add cipd support to gclient.	7 years ago
gclient_utils_test.py	gclient: Stop printing newline in front of "________ running hook"	8 years ago
git_cache_test.py	Allow 'git cache fetch' to re-bootstrap.	10 years ago
git_cl_creds_check_report.txt	git cl creds-check: improve report and give better recommendation.	8 years ago
git_cl_test.py	git-cl-patch: fail if patching from a different repo	7 years ago
git_common_test.py	Make git blame test pass independently of core.abbrev setting	8 years ago
git_dates_test.py	Added git hyper-blame, a tool that skips unwanted commits in git blame.	9 years ago
git_drover_test.py	Reland "Use core.quotePath=false when git is listing files"	8 years ago
git_footers_test.py	Let git_footers split final paragraphs in specific cases	7 years ago
git_hyper_blame_test.py	GitHyperBlameMainTest.testBadFilename: Work around git-blame bug.	8 years ago
git_number_test.py	Disable tests that break because of git 2.4	10 years ago
git_rebase_update_test.py	Add better error messages to git_reparent_branch.	10 years ago
gsutil_test.py	download_from_google_storage: Fix tests and rename	8 years ago
owners_finder_test.py	Only attribute comments to owners that are close to the comment	8 years ago
owners_unittest.py	Also check override files before checking the fs	8 years ago
patch_test.py	Fix applying svn:executable on Windows.	13 years ago
presubmit_unittest.py	[presubmit] Remove line	8 years ago
recipes_test.py	Stop running deprecated simulation_test	8 years ago
scm_unittest.py	Correct Git show's path format on Windows	8 years ago
subprocess2_test.py	depot_tools: Replace pylint error numbers with symbolic names.	9 years ago
upload_to_google_storage_unittest.py	tests: Use the right clean up method name for some test cases.	8 years ago
watchlists_unittest.py	depot_tools: Replace pylint error numbers with symbolic names.	9 years ago