You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
8254f06ba7
The multiprocessing module on Windows has a bug where if you ask for more than 60 child processes then it will hang. This is related to the MAXIMUM_WAIT_OBJECTS (64) limit of WaitForMultipleObjects. Other sources have listed the multiprocessing limit as being 61, or have said that the the maximum number of objects that can be waited on is actually 63, but those details don't really matter. The original fix for this class of issues was crrev.com/c/2785844. This change extends those fixes to depot_tools, which was missed last year. This change also updates how PyLint is called by further limiting the number of jobs to the number of files being processed divided by 10. This is because there is a significant cost to creating PyLint subprocesses - each takes about 0.5 s on my test machine. So there needs to be enough parallelism to justify this. Patches for PyLint and a bug for cpython are planned. This will stop PyLint from hanging during presubmits on many-core machines. The command used to reproduce the hangs and validate the fix was: git cl presubmit -v --force --files "chrome/test/mini_installer/*.py" Prior to this change this command would use (on my many-core test machine) 96 processes and would hang. How it uses just two processes because there are only 16 files to analyze. Output before: Pylint (16 files using ['--disable=cyclic-import'] on 96 cores) Output after: Pylint (16 files using ['--disable=cyclic-import'] on 2 processes) This is actually not quite true because the hang would prevent the old message from being displayed. Bug: 1190269, 1336854 Change-Id: Ie82baf91df4364a92eb664a00cf9daf167e0a548 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3711282 Reviewed-by: Gavin Mak <gavinmak@google.com> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> |
3 years ago | |
|---|---|---|
| .. | ||
| gstools | ||
| OWNERS | 3 years ago | |
| PRESUBMIT.py | 4 years ago | |
| auth_test.py | 5 years ago | |
| autoninja_test.py | 3 years ago | |
| bot_update_coverage_test.py | 3 years ago | |
| cipd_bootstrap_test.py | 5 years ago | |
| detect_host_arch_test.py | 4 years ago | |
| download_from_google_storage_unittest.py | 3 years ago | |
| fetch_test.py | 3 years ago | |
| fix_encoding_test.py | 4 years ago | |
| gclient_cipd_smoketest.py | 3 years ago | |
| gclient_eval_unittest.py | 4 years ago | |
| gclient_git_mutates_smoketest.py | 4 years ago | |
| gclient_git_smoketest.py | 4 years ago | |
| gclient_paths_test.py | 5 years ago | |
| gclient_scm_test.py | 3 years ago | |
| gclient_smoketest.py | 5 years ago | |
| gclient_smoketest_base.py | 3 years ago | |
| gclient_test.py | 3 years ago | |
| gclient_transitions_smoketest.py | 5 years ago | |
| gclient_utils_test.py | 3 years ago | |
| gerrit_client_test.py | 4 years ago | |
| gerrit_util_test.py | 4 years ago | |
| git_cache_test.py | 5 years ago | |
| git_cl_creds_check_report.txt | 5 years ago | |
| git_cl_test.py | 3 years ago | |
| git_common_test.py | 3 years ago | |
| git_dates_test.py | 6 years ago | |
| git_find_releases_test.py | 3 years ago | |
| git_footers_test.py | 4 years ago | |
| git_hyper_blame_test.py | 5 years ago | |
| git_map_test.py | 4 years ago | |
| git_migrate_default_branch_test.py | 3 years ago | |
| git_number_test.py | 6 years ago | |
| git_rebase_update_test.py | 3 years ago | |
| gsutil_test.py | 4 years ago | |
| lockfile_test.py | 5 years ago | |
| metrics_test.py | 4 years ago | |
| my_activity_test.py | 4 years ago | |
| ninjalog_uploader_test.py | 3 years ago | |
| owners_client_test.py | 4 years ago | |
| owners_finder_test.py | 4 years ago | |
| owners_unittest.py | 5 years ago | |
| presubmit_unittest.py | 3 years ago | |
| rdb_wrapper_test.py | 5 years ago | |
| recipes_test.py | 4 years ago | |
| roll_dep_test.py | 5 years ago | |
| scm_unittest.py | 4 years ago | |
| split_cl_test.py | 4 years ago | |
| subcommand_test.py | 5 years ago | |
| subprocess2_test.py | 3 years ago | |
| upload_to_google_storage_unittest.py | 4 years ago | |
| utils_test.py | 3 years ago | |
| watchlists_unittest.py | 5 years ago | |