Pylint uses parallelism to improve performance, but high startup costs
means that this only makes sense if there are lots of files to be
processed. So, a while ago a change was made such that if there are
fewer than ten files to be analyzed then no parallelism is used.
Our Pylint wrapper also has a hack where it does one type of check in
serial mode, because that is the only time it is reliable. This
requires running Pylint twice, which is expensive.
If there are a small enough number of files to analyze then we will be
doing serial analysis anyway, so there is no need to do two separate
runs. In this test case:
git cl presubmit -v --files tools\code_coverage\create_js_source_maps\test\create_js_source_maps_test.py
the cost of Pylint is dropped roughly in half, from six seconds to
three seconds, by eliminating one of the three-second runs.
Bug: 1309977
Change-Id: I2e5e96a86d1d76b127f481af7478d807c042b609
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3812436
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This update is per http://crrev.com/1031130 where OSS licensing
requested removal of "All rights reserved" from new files.
Change-Id: I52a9deab99539ea90403798606c7bbeb261df994
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3811744
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Robert Liao <robliao@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Recent gerrit issues made it obvious that "git cl presubmit" relies on
gerrit much more than most people would expect (the expectation is zero
for many people). This makes presubmits flaky or much slower under poor
network conditions, and it means that the presubmit step may drastically
underestimate how long it takes to run because of a
cl.FetchDescription() that may occur outside of the timed portion of the
presubmits.
This change wraps more network-touching steps in try/except blocks, to
make them robust. It also gets them to check for the existence of a
PRESUBMIT_SKIP_NETWORK environment variable. And, it prints the elapsed
time to get the CL description if this is inordinately long.
Bug: 1350227
Change-Id: I7954fd50e928fd24975a4f61a316cb280542ebbd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3813095
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Change crrev.com/c/3788227 fixed one instance of errors being treated as
warnings when running "git cl presubmit" with the --upload option. This
is undesirable when testing "git cl presubmit" with --all or --files,
because it makes the errors harder to find. This change fixes four
more newly discovered instances of this behavior.
That is, this change makes it so that pylint issues and other serious
problems will be reported as errors when running:
git cl presubmit --force --all --upload
This will make the pylint errors that this command triggers easier to
find and fix:
git cl presubmit --force --upload --files mojo\public\tools\bindings\*.py
This change does _not_ turn cpplint warnings into errors, even though
they are errors when running non-upload presubmits. That is because
there are several directories that only run cpplint on upload and these
directories have many errors and there is no short-term path to changing
this.
Bug: 1309977
Change-Id: If49f820fc6894dcd1d9aaaf4d932b04f79922bc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3791744
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
When upload presubmits are being run then GetUnitTests would treat
errors as warnings, so as to not stop developers from uploading their
changes. This makes sense when patches are being manually uploaded, but
doesn't make sense when running a "git cl presubmit --all --upload" bot.
This behavior caused nine failures (but not all failures) to be missed
when doing these tests.
This change makes these failures errors when testing with --all
--upload, but leaves them as warnings for other --upload purposes.
Bug: 1309977
Change-Id: Ibf149475e4cdee10bbbbc86fd0ab668b7a679089
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3788227
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
If a presubmit test is misconfigured then it may end up running on
neither Python 2 nor Python 3. This was supposed to trigger a warning
but the warning code was broken for almost a year. It was then fixed so
that it issued an error, but that caused some breakage, so it was made a
warning.
The known problems with tests not running at all have been fixed and
the warning has been on for over two weeks so this change turns the
warning back into an error. This is appropriate because a test that
is not run at all is a serious bug. This change will allow us to move
more confidently when switching tests to Python 3, and it shouldn't
cause any additional breakage.
Bug: 1330859
Change-Id: I51028bb9c896c60c5cf3ccb6f472ade0bb1e0c16
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3717242
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
The multiprocessing module on Windows has a bug where if you ask for
more than 60 child processes then it will hang. This is related to the
MAXIMUM_WAIT_OBJECTS (64) limit of WaitForMultipleObjects. Other sources
have listed the multiprocessing limit as being 61, or have said that the
the maximum number of objects that can be waited on is actually 63, but
those details don't really matter.
The original fix for this class of issues was crrev.com/c/2785844. This
change extends those fixes to depot_tools, which was missed last year.
This change also updates how PyLint is called by further limiting the
number of jobs to the number of files being processed divided by 10.
This is because there is a significant cost to creating PyLint
subprocesses - each takes about 0.5 s on my test machine. So there needs
to be enough parallelism to justify this.
Patches for PyLint and a bug for cpython are planned.
This will stop PyLint from hanging during presubmits on many-core
machines. The command used to reproduce the hangs and validate the fix
was:
git cl presubmit -v --force --files "chrome/test/mini_installer/*.py"
Prior to this change this command would use (on my many-core test
machine) 96 processes and would hang. How it uses just two processes
because there are only 16 files to analyze.
Output before:
Pylint (16 files using ['--disable=cyclic-import'] on 96 cores)
Output after:
Pylint (16 files using ['--disable=cyclic-import'] on 2 processes)
This is actually not quite true because the hang would prevent the
old message from being displayed.
Bug: 1190269, 1336854
Change-Id: Ie82baf91df4364a92eb664a00cf9daf167e0a548
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3711282
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
presubmit --all should succeed regardless of the current state of the
tree. Otherwise a presubmit --all bot will be flaky due to circumstances
beyond its control.
This is a logical follow-on to crrev.com/c/3628368 which turned some
other errors into warnings for presubmit --all.
Bug: 1309977
Change-Id: I6d26708f8c47916cf015dec75e27833db7ca465b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3696070
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
A run of "git cl presubmit --all" shows that PyLint 1.5 is being run
from nine locations, however the deprecation warnings don't say where.
This makes it difficult to file bugs or fix the remaining instances.
This changes the message to list the path to the presubmit that is
running PyLint 1.5.
This also adds instructions on how to change to version 2.7.
Before:
pylint-1.5 is deprecated, please switch to 2.7 before 2022-07-11
After (with word wrapping):
pylint-1.5 is being run on ...src\tools\find_runtime_symbols
and is deprecated, please switch to 2.7 before 2022-07-11 (add
version='2.7' to RunPylint call)
Change-Id: Iece2cb904f5d26ad66e3ab78f7ce7aef1878bfd1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688839
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When PRESUBMIT is run with a custom --upstream, dry run cl format should
also use this value, otherwise format warnings would be inconsistent.
Change-Id: I1370f614618ecc03706e54cd4b151a207d9b3bae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3688631
Auto-Submit: Aleksey Khoroshilov <akhoroshilov@brave.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 013e47be09.
Reason for revert: This will be relanded as a warning, although the
bugs which the original change exposed should be fixed soon.
Original change's description:
> Revert "Fix not-run-test detection."
>
> This reverts commit 41691abe86.
>
> Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.
>
> Original change's description:
> > Fix not-run-test detection.
> >
> > In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> > the problem of some tests being run neither on Python 2 or Python 3. At
> > the same time reporting was added to warn if tests were run on neither
> > Python version.
> >
> > Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> > object was created but was not added to the list of results, so it had
> > no effect. Also the skipped tests name was added where a list was
> > expected which meant that each character of the test name showed up on
> > its own line.
> >
> > This change fixes the reporting. It also changes the format of the
> > report and makes it an error - a failure to run tests at all deserves
> > that severity level.
> >
> > A test run with the fixed check showed that no errors have crept in -
> > all tests were being run with one or the other. However the fix did
> > find a bug in an in-progress change I was working on.
> >
> > Bug: 1223478
> > Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> > Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> > Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> > Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
>
> Bug: 1223478
> Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Bug: 1223478, 1330859
Change-Id: I6a0a78e70db77c9c5fda5bab27960cd4e71279e5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685349
Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This reverts commit c03ebf0e1a.
Reason for revert: the original change was not the cause of the test
failures. The actual cause was crrev.com/c/3683105 which was reverted.
Original change's description:
> Revert "Add Python 3 support to GetPythonUnitTests"
>
> This reverts commit dfc71bbe01.
>
> Reason for revert: Breaks all presubmits that do not have skip_shebang_check=True set.
>
> Original change's description:
> > Add Python 3 support to GetPythonUnitTests
> >
> > GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
> > scripts using Python 2. This change adds a python3=False parameter to
> > GetPythonUnitTests to give the option of switching to Python 3, while
> > leaving the default as Python 2. The child scripts are run under one or
> > the other, based on this parameter.
> >
> > This change is needed in support of crrev.com/c/3679801.
> >
> > Bug: 1313804
> > Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
> > Reviewed-by: Jesse McKenna <jessemckenna@google.com>
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
>
> Bug: 1313804
> Change-Id: I1bd7096b6cfdfed6205e6dc0b5d4498b5b821b97
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Christoffer Jansson <jansson@chromium.org>
> Owners-Override: Ben Pastene <bpastene@chromium.org>
> Reviewed-by: Tomas Gunnarsson <tommi@chromium.org>
Bug: 1313804
Change-Id: I15ed90cc35d88ece71eb681acc0acc06cb5c3bcc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3685229
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
This reverts commit 41691abe86.
Reason for revert: I thought https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378 would solve the problem but turns out this is the issue where it now reports it as a critical error.
Original change's description:
> Fix not-run-test detection.
>
> In crrev.com/c/2986400 the skip_shebang_check flag was added to address
> the problem of some tests being run neither on Python 2 or Python 3. At
> the same time reporting was added to warn if tests were run on neither
> Python version.
>
> Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
> object was created but was not added to the list of results, so it had
> no effect. Also the skipped tests name was added where a list was
> expected which meant that each character of the test name showed up on
> its own line.
>
> This change fixes the reporting. It also changes the format of the
> report and makes it an error - a failure to run tests at all deserves
> that severity level.
>
> A test run with the fixed check showed that no errors have crept in -
> all tests were being run with one or the other. However the fix did
> find a bug in an in-progress change I was working on.
>
> Bug: 1223478
> Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
> Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
> Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
> Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
Bug: 1223478
Change-Id: If4395e90784c3f91692f398b4e5230a51139f78a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683105
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
This reverts commit dfc71bbe01.
Reason for revert: Breaks all presubmits that do not have skip_shebang_check=True set.
Original change's description:
> Add Python 3 support to GetPythonUnitTests
>
> GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
> scripts using Python 2. This change adds a python3=False parameter to
> GetPythonUnitTests to give the option of switching to Python 3, while
> leaving the default as Python 2. The child scripts are run under one or
> the other, based on this parameter.
>
> This change is needed in support of crrev.com/c/3679801.
>
> Bug: 1313804
> Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
> Reviewed-by: Jesse McKenna <jessemckenna@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Bug: 1313804
Change-Id: I1bd7096b6cfdfed6205e6dc0b5d4498b5b821b97
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3683378
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Christoffer Jansson <jansson@chromium.org>
Owners-Override: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Tomas Gunnarsson <tommi@chromium.org>
GetPythonUnitTests (and by extension RunPythonUnitTests) always ran
scripts using Python 2. This change adds a python3=False parameter to
GetPythonUnitTests to give the option of switching to Python 3, while
leaving the default as Python 2. The child scripts are run under one or
the other, based on this parameter.
This change is needed in support of crrev.com/c/3679801.
Bug: 1313804
Change-Id: Ic59287352d4941707adaf7981ed7af4201b7d526
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3680099
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
In crrev.com/c/2986400 the skip_shebang_check flag was added to address
the problem of some tests being run neither on Python 2 or Python 3. At
the same time reporting was added to warn if tests were run on neither
Python version.
Unfortunately the reporting code was incorrect. A PresubmitPromptWarning
object was created but was not added to the list of results, so it had
no effect. Also the skipped tests name was added where a list was
expected which meant that each character of the test name showed up on
its own line.
This change fixes the reporting. It also changes the format of the
report and makes it an error - a failure to run tests at all deserves
that severity level.
A test run with the fixed check showed that no errors have crept in -
all tests were being run with one or the other. However the fix did
find a bug in an in-progress change I was working on.
Bug: 1223478
Change-Id: Ibb44b5e60e7a7a5de08302f19ee4035cdfac5212
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3674199
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
Reviewed-by: Fabrice de Gans <fdegans@chromium.org>
A typical cpplint error report during a presubmit looks like this:
.../linux_ui.cc:64: Almost always, snprintf is better than strcpy [runtime/printf] [4]
...
Changelist failed cpplint.py check.
The problem is that the details of the error go to stderr when
CheckChangeLintsClean is run, whereas the "Changelist failed cpplint.py
check." message is printed at the end of the run. If there a lot of
errors, warnings, or messages then the association between them can be
completely lost.
This change adds the tag (cpplint) to the error messages, and adds a
comment to the "failed cpplint.py check" message to suggest looking for
the tag.
Bug: 1309977
Change-Id: I0d073b57f215e28495cbc3e009cab6ac2f23b65e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3632947
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Some of the expensive checks when running presubmit --all, such as
CheckStableMojomChanges (~300 s) and CheckAddedDepsHaveTargetApprovals
(~200 s) only look at diffs and are therefore guaranteed to be NOPs when
running presubmit --all or --files=. Passing along the no_diffs state
lets these expensive checks be skipped, thus allowing for faster
iteration times.
Initial testing suggests that (with some supporting changes in the
Chromium repo) this reduces "presubmit --all" times by about ten
minutes, or a bit more than 10%, and additional improvements may be
possible.
Special handling for the no-diffs case also offers a simple way to avoid
presubmit failures that happen whenever all files are flagged as being
changed.
Finally, and perhaps most importantly for having a presubmit --all bot,
when --no_diffs is passed we can treat errors like "Issue wasn't
uploaded" and "Add a description to the CL" as messages, thus making it
possible to have zero presubmit errors when run on origin/main.
Bug: 1320937, 1322936
Change-Id: I0d09dd4aae8fdaa48c8b2f89337441cf96dcff72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3628368
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
While enabling cpplint on base I found an extraneous error and an overly
complicated invocation of CheckChangeLintsClean. This change disables
the extraneous error and improves the file filtering defaults to make it
easier to invoke cpplint.
The warning was:
base\win\win_util.cc(193): Missing username in TODO; it should look
like "// TODO(my_username): Stuff." [readability/todo] [2]
Chromium has over 10,000 TODO comments that are links to bugs instead of
users so we clearly want to suppress this warning.
Most calls to CheckChangeLintsClean in Chromium pass in custom file
filtering in order to only analyze C++ files. Failing to pass a file
filter leads to errors like this:
Ignoring base\win\embedded_i18n\create_string_rc.py; not a valid
file name (cc, cuh, cu, cpp, h)
Ignoring c:\src\chromium\src\components\arc\PRESUBMIT.py; not a
valid file name (cu, cc, cuh, cpp, h)
It doesn't make sense for the canned checks version of
CheckChangeLintsClean to default to passing in files that cpplint will
complain about so if source_file_filter is None then I configure an
appropriate filter.
This change also switches to vs7 output format by default on Windows,
and deletes a trailing space.
Bug: 1309977
Change-Id: I342ca9e56a41502755dfc09307b441dfa6a48ec0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3655592
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
PanProjectChecks is called twice, once from the root PRESUBMIT.py and
once from third_party/blink/PRESUBMIT.py. For the file-based checks a
series of filters avoids redundant reporting, but certain global checks
were being run twice, potentially causing duplicate reporting.
This adds an optional global_checks flag which can be set to false in
the blink PRESUBMIT.
Bug: 1309977
Change-Id: Ia43a7a0f29cf7b58ac9bab0e9aa374e9bf0155a3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3611409
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Jesse McKenna <jessemckenna@google.com>
Header guards are checked by CheckForIncludeGuards which has broader
coverage than cpplint.py and should therefore be preferred.
Having two include-guard checkers that cover different sets of files and
use different tags to silence warnings is just confusing.
Bug: 1309977
Change-Id: Ic0d1ad610cf9c34d6777d852da2e2e22b8686552
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587725
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
When CheckLongLines finds problems it prints the first five, but this
gives no indication as to whether there are just five, or hundreds.
This change prints the number of long lines found.
The snapshot function in PanProjectChecks prints the elapsed time for
long checks, however it has two issues that make it inconvenient. One is
that it prints the time in ms which sounds great until you get a warning
that one of the checks is taking 2041373 ms (not kidding, although now
fixed) which is tricky to read. The other problem is that it was
actually measuring CPU time, not wall-clock time. This changes it to
print the time in seconds (to 0.1 seconds) and to measure elapsed
time.
Bug: 1309977
Change-Id: I7564a8cdf7bb3349b10ebbddbfe179188d4bf309
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3587726
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
After optimizing CheckForIncludeGuards the CheckLicense presubmit was
the second slowest presubmit, taking about 35 minutes when run as part
of "git cl presubmit --all". The cause of its slowness was initially
non-obvious because it runs quite quickly on most files, however
analysis showed that it could take 50+ seconds on some files. The files
that it is slow on are those that lack a good license header, meaning
that the regex match has to painstakingly scan the entire file.
The optimization in this change is to recognize that there is a simple
non-regex line that appears in all valid license headers, regardless of
variants. If that line is absent then there is, necessarily, no valid
license header, and searching for a line of text is something that
Python can do extremely quickly.
This change drops the CheckLicense time from about 35 minutes to about
32 seconds.
Trivia: _CommonChecks in third_party/blink/PRESUBMIT.py passes .* as the
license, so I added an early-out for that to avoid pointlessly scanning
those files.
Bug: 1309977
Change-Id: Ic2e56079675c2c5a2643d20dd492b1cc52e4ead2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3584882
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
CheckDirMetadataFormat executes dirmd on all applicable files. When
running "git cl presubmit --all" that ends up being ~7,500 files and
the command line that is generated is ~500,000 characters. Windows does
not like that.
This change breaks up the check into multiple invocations of dirmd in
order to avoid these limits. This is important because running all
presubmits is the only way to systematically find bugs, and avoid being
surprised by them when submitting a change.
Bug: 1309977
Change-Id: I24fbc340cdb975dbe7f6a2132e516d6f7e2f9165
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3554633
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
These lints are redundant and in rare cases may conflict with
clang-format output. We treat clang-format's output as the correct
formatting, so disable the lints.
The following whitespace lints are retained, all others are disabled:
whitespace/blank_line
whitespace/comments
whitespace/empty_conditional_body
whitespace/empty_if_body
whitespace/empty_loop_body
whitespace/ending_newline
whitespace/todo
Bug: chromium:1240500
Change-Id: I22e2d8ea4de021ea389e155d8c9e096c3ae6f8b7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3231485
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Collin Baker <collinbaker@chromium.org>
Chrome uses the syntax "Bug: xyz" in commit messages to attribute CLs
to bugs while Critique uses "BUG=xyz". As per
```
git log --grep '\(Bug\|Fixed\)=' --since=2021-01-01 --format=oneline \
| wc -l
```
we've had 27 CLs this year using the wrong syntax which led to crbug.com
not learning about the CLs <-> Bug attribution. Yours truly caused one
of them and wants to prevent the problem in the future.
We might relax the requirments in crbug.com or
chromium-review.googlesource.com but then anyone grepping through the
commit logs would need to deal with the ambiguity. Therefore, we can
just enforce the right syntax here.
https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/contributing.md
contains the correct syntax.
Change-Id: I60ee579deac50a74e1a014ceb1d9744cbc10ab41
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3141567
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Dominic Battré <battre@chromium.org>
vpython doesn't parse shebang. If python version is not explicitly set
in vpython file, version 2 will be used. pylint 2.6 requires py3.
R=dpranke@google.com
Bug: 1242737
Change-Id: I21a3c4b26c24860b4c8fd07c6d5cc6d5149af2fc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3120321
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
We were attempting to run pylint-2.6 with python2, when
it needs to be run with python3. I'm not sure how we
haven't noticed this before now.
Bug: 1157676
Change-Id: I60fc22a805c20649d04f7d0f53c828a0db41e8e7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3068833
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Peter Wen <wnwen@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
If user enters buganizer bug prefixed with b/, warn user that b: is
expected.
R=apolito@google.com
Bug: 1226474
Change-Id: Ifbced843f9eecc00560cc273ae06bff9d435a815
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3016587
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
JSON check may be useful for files that don't necessaraly end with
.json. For example, infra/config/recipes.cfg should be a json file. This
change allows users to add custom file_filter that can include such
files.
R=apolito@google.com
Bug: 1223923
Change-Id: Ia6fc7f86fa368510baaee978d9a0a27eccb6b31f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2992735
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This allows us to migrate PRESUBMIT.py files over to pylint-2.6 bit by
bit. It also allows us to support and upgrade pylint to newer versions
in the future.
Bug: 1221143
Fixed: 1221143
Change-Id: I9af96f5f06caf48e9923ad5fae75b98a0a7aeb49
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2994723
Commit-Queue: Peter Wen <wnwen@chromium.org>
Auto-Submit: Peter Wen <wnwen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
A bug in the logic enabling tests to be run on python3 resulted in
tests never being run in python3 unless they also have a shebang line
pointing to python3. Since this bug was introduced, effectively, most
tests that are supposed to be run on python3 do not run at all. Since
then a lot of python3 tests have regressed in Chromium.
In order to fix the issue moving forward, this introduces a new
argument that skips the shebang check to run a test in python3. Tests
will be re-enabled one by one until every single instance in Chromium
has been updated.
Bug: 1223478
Change-Id: I91a0688c6f4d9b4fbf18e3d446366cded8c7f2f1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2986400
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
If you copy\paste a license header in such a way that \r\n line endings
are used (I have done this) then CheckLicense would fail. This is quite
confusing because the license looks perfect in most text editors. This
change tells CheckLicense to treat \r\n line endings as equivalent to \n
and let CheckForWindowsLineEndings do its job of warning about the \r\n
line endings. This avoids confusing presubmit messages.
Bug: 801033
Change-Id: I37fa4a6d9cd0f1a4dcce1267175f4b8fd298b906
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2951788
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
kwargs['stdin'] is passed in to subprocess communicate(), which
expects bytes, not str.
Bug: 1215602
Change-Id: I95326eed08cffecd31abc6b0380de2a3c2ff3dd6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2929386
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Dan H <harringtond@chromium.org>
The CheckLicense presubmit is supposed to check for correct licenses,
not for correct line endings. By reading the input file in binary mode
it implictly does both, leading to confusing error messages.
This is exacerbated by bugs in the line ending checks which are being
addressed separately. This change will avoid false positives on files
that have Windows line endings.
Change-Id: I2ff8632f273ec4896cb4918c386e0d1c12e72935
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2910486
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This will attempt to do so in a Python2-compatible way.
Bug: 1210746
Change-Id: I09edc21a5c47106803c0ac5ca449e0f8732efb24
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2906501
Auto-Submit: Dirk Pranke <dpranke@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Gavin Mak
Josip Sokcevic
Bruce Dawson
Robert Liao
Aleksey Khoroshilov
Christoffer Jansson
Aravind Vasudevan
Takuto Ikuta
Collin Baker
Dominic Battre
Dirk Pranke
Johann
Peter Wen
Fabrice de Gans
Dan Harrington