You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1f067b88df
The first call to gsutil causes a network call and makes the download script slow also in the most optimal cases. This CL refactors the download script and moves the first gsutil call after checking locally if sha1s match. 1) This turns the input acquisition into a generator and buffers the files and sha1s in a list before multithreading. 2) This sequentially checks the sha1s and files and bails out early if all match. In Chrome-land, we usually call this script with only one file. There are some cases with around 4. This could also be parallelized if the need arises. 3) The initial gsutil check, which ensures gsutil is updated, is moved right in front of the multithreaded downloads. The performance of one call to download_from_google_storage for an existing 500MB file is 2.3s before this CL and 1.2s after (most of the remaining time left is spent for making sha1sum). Example for full gclient runhooks (when everything is up-to-date): Chromium: 32s before, 12s after V8: 12s before, 3s after Bug: 776311 Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317 Reviewed-on: https://chromium-review.googlesource.com/897562 Commit-Queue: Michael Achenbach <machenbach@chromium.org> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Reviewed-by: Ryan Tseng <hinoka@chromium.org> |
7 years ago | |
|---|---|---|
| .. | ||
| gstools | ||
| auth_test.py | 8 years ago | |
| bot_update_coverage_test.py | ||
| checkout_test.py | ||
| download_from_google_storage_unittest.py | 7 years ago | |
| fix_encoding_test.py | ||
| gclient_eval_unittest.py | ||
| gclient_scm_test.py | 7 years ago | |
| gclient_smoketest.py | 7 years ago | |
| gclient_test.py | 7 years ago | |
| gclient_utils_test.py | ||
| git_cache_test.py | ||
| git_cl_creds_check_report.txt | ||
| git_cl_test.py | 7 years ago | |
| git_common_test.py | ||
| git_dates_test.py | ||
| git_drover_test.py | 7 years ago | |
| git_footers_test.py | 7 years ago | |
| git_hyper_blame_test.py | ||
| git_number_test.py | ||
| git_rebase_update_test.py | ||
| gsutil_test.py | ||
| owners_finder_test.py | ||
| owners_unittest.py | 8 years ago | |
| patch_test.py | ||
| presubmit_unittest.py | 8 years ago | |
| recipes_test.py | ||
| scm_unittest.py | ||
| subprocess2_test.py | ||
| upload_to_google_storage_unittest.py | ||
| watchlists_unittest.py | ||