Commit Graph

404 Commits (7958e3057968173e26988d921ee844d80e8ebdf7)

Author SHA1 Message Date
Josip Sokcevic 7958e30579 Revert "Move scm.py to lib/"
This reverts commit 688adfe276.

Reason for revert: breaks try jobs

Original change's description:
> Move scm.py to lib/
>
> It belongs there.
>
> Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

Change-Id: I662907b8bb4b2e7db82409bd34c3905db7961a12
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4300261
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Josip Sokcevic 688adfe276 Move scm.py to lib/
It belongs there.

Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 512fd3bd85 Extract searching relevat files in utils function
This change introduces lib/ directory which will be used for classes
that are considered libraries and used by multiple executables.

Change-Id: I3da126778bf5ffdb28d7a3c2b966c85de08ba717
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289679
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Brian Ryner 790a0c522d Fix CheckCIPDManifest for python 3.
Change-Id: Icf7b8e7ad732a731520550ff9e276f13016effb7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4149964
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Brian Ryner <bryner@google.com>
2 years ago
Pavol Marko 624e7eec34 Change working directory to PRESUBMIT's dir also for postsubmit
While we're at it, ensure that the working directory is reverted even if
exceptions are raised by moving that part into a finally block.

Bug: 1404222
Change-Id: I8606bc5810202ff1ecdc7f202e3f574bf643f91d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4129515
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Pavol Marko <pmarko@chromium.org>
2 years ago
Bruce Dawson 7a81ebf111 Add .rs to list of presubmit source file patterns
.rs files were not part of DEFAULT_FILES_TO_CHECK which means that they
were not being checked for proper licenses, among other things.

Testing with this command reveals that six missing or bad licenses have
crept in:

    git cl presubmit --force --files "*.rs"

Output was:

Found a bad license header in these files:
  build\rust\tests\test_aliased_deps\main.rs
  build\rust\tests\test_aliased_deps\real_name.rs
  mojo\public\rust\tests\util\mojom_validation.rs
  testing\rust_gtest_interop\gtest_attribute.rs
  tools\crates\gnrt\sample_package\foo\src\lib.rs
  tools\crates\gnrt\sample_package\src\main.rs

Change-Id: Ia7788d92d428a4060f4a0b854fdcee0450b0a3f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4128762
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Collin Baker <collinbaker@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Bruce Dawson 31ca877242 Fix CheckLicense tests
A last minute change to the CheckLicense tests in crrev.com/c/4032366
used the _GetLicenseText function in the wrong place, thus neutering one
test and duplicating code in another. This fixes the tests so that they
minimize duplication and actually test the CSS/C style comments.

Change-Id: Ibbe880dec8d1e85416ed1b7ce94922c6f2c2a543
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4037103
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2 years ago
Bruce Dawson 25cf78395c Fix two CheckLicense bugs and improve tests
Some recent "improvements" to CheckLicense were buggy, at least
partially because I did not increase test coverage when adding new code
paths. This change fixes two known mistakes and greatly increases test
coverage.

One bug was that the code assumed that match.groups()[0] would always be
the current year, but that assumption is invalid when a license regex is
passed in.

Another bug is that the new_license_re took away the possibility of the
license ending with */ but that is used in .css files.

The increased test coverage catches these and validates other
assumptions around new files.

The failures were reproduced and then the fixes validated using this
command:

  vpython3 tests/presubmit_unittest.py

Bug: 1098010
Change-Id: Ic91ca9f7a182660aef7b1eead79e841f58e15a3e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4032366
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2 years ago
Erik Staab 9f38b63b4e Add snippets to presubmit failures in resultdb
Fixed: 1174446
Change-Id: Ie57818846704ed24fe1af49a70e24ad7a118d44c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3965229
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Erik Staab <estaab@chromium.org>
3 years ago
Gavin Mak ff5d25bb10 Add canned presubmit checks for corp links
Only googlesource links should be used publicly.

Bug:b/253074243
Change-Id: Id972821acbfde635717e21bba879880e5312f92c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3955706
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Fabrice de Gans 9ebcfa6be1 [code-health] Run CheckVPythonSpec under python3
This change also makes CheckVPythonSpec() look for .vpython3 files.

Bug: 1336295
Change-Id: I4bbec48debe42748811a8cfcef6da9602017d4bb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3891974
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Fabrice de Gans <fdegans@chromium.org>
3 years ago
Bruce Dawson 443f9851a4 Improve Python 2 presubmit warning
The warning that a PRESUBMIT.py file was still being run under Python 2
was apparently too polite as one was just added. This change adds
instructions on how to prevent this, which should make it clear that
the behavior is not desirable.

The warning was added in crrev.com/c/3826917.

Bug: 1313804
Change-Id: I81a6ea45907b5ea00e7d945c3131a329708cc377
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3878875
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Gavin Mak 0f1addcbe9 Reland "Remove old DepotToolsOwners implementation"
This is a reland of commit 10dbd7babd

Original change's description:
> Remove old DepotToolsOwners implementation
>
> code-owners should have been enabled for most hosts that depot_tools
> supports by now. Remove our own implementation and rely on code-owners.
>
> Change-Id: Iaf0d3db65b2e5063b67d42b92188c4ec51d2cd9a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3783475
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Change-Id: Ic87f34411bb38b91fcf49adb07601ae10244e828
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3881043
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
3 years ago
Josip Sokcevic 6efb3ed962 Reland "Use io.open for opening files"
This is a reland of commit eb16430cf9

Original change's description:
> Use io.open for opening files
>
> Use io.open which is consistent in py2 and py3.
>
> Bug: 1357152
> Change-Id: I49a3df503026bc6918362a9b5600f57111111111
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3872429
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1357152
Change-Id: I58881c040ca7333f0247875a0c4abe31cf793b08
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3877645
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
3 years ago
Gavin Mak a3a95d439b Revert "Use io.open for opening files"
This reverts commit eb16430cf9.

Reason for revert: blocking upload of CLs, https://crbug.com/1357152#c8

Original change's description:
> Use io.open for opening files
>
> Use io.open which is consistent in py2 and py3.
>
> Bug: 1357152
> Change-Id: I49a3df503026bc6918362a9b5600f57111111111
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3872429
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Bug: 1357152
Change-Id: Iec18fe8fd37f28887507d47d7b35b93ccee72375
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3872487
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
3 years ago
Josip Sokcevic eb16430cf9 Use io.open for opening files
Use io.open which is consistent in py2 and py3.

Bug: 1357152
Change-Id: I49a3df503026bc6918362a9b5600f57111111111
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3872429
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Bruce Dawson 8d78bd1fc1 Print which PRESUBMIT.py scripts run on Python 2
When running presubmits you can tell if Python 2 presubmits are running,
but it's subtle, and it isn't obvious which presubmits are running under
Python 2.

This makes it obvious by always printing the path to PRESUBMIT.py
scripts that are run under Python 2. This change also fixes a bug in the
--verbose logic - it would print that a script was running before
evaluating _ShouldRunPresubmit.

Bug: 1352347
Change-Id: I45213acdb3b37f7be0713e5caba775841704817c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3826917
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Bruce Dawson 0af709772e Don't do double-serial Pylint checks
Pylint uses parallelism to improve performance, but high startup costs
means that this only makes sense if there are lots of files to be
processed. So, a while ago a change was made such that if there are
fewer than ten files to be analyzed then no parallelism is used.

Our Pylint wrapper also has a hack where it does one type of check in
serial mode, because that is the only time it is reliable. This
requires running Pylint twice, which is expensive.

If there are a small enough number of files to analyze then we will be
doing serial analysis anyway, so there is no need to do two separate
runs. In this test case:

  git cl presubmit -v --files tools\code_coverage\create_js_source_maps\test\create_js_source_maps_test.py

the cost of Pylint is dropped roughly in half, from six seconds to
three seconds, by eliminating one of the three-second runs.

Bug: 1309977
Change-Id: I2e5e96a86d1d76b127f481af7478d807c042b609
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3812436
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Gavin Mak a9677a5e9b Revert "Remove old DepotToolsOwners implementation"
This reverts commit 10dbd7babd.

Reason for revert: removal broke builders

Original change's description:
> Remove old DepotToolsOwners implementation
>
> code-owners should have been enabled for most hosts that depot_tools
> supports by now. Remove our own implementation and rely on code-owners.
>
> Change-Id: Iaf0d3db65b2e5063b67d42b92188c4ec51d2cd9a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3783475
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Change-Id: I781648f470e8f2c9d408ae7112cfc41060f5385d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3818251
Auto-Submit: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Gavin Mak 10dbd7babd Remove old DepotToolsOwners implementation
code-owners should have been enabled for most hosts that depot_tools
supports by now. Remove our own implementation and rely on code-owners.

Change-Id: Iaf0d3db65b2e5063b67d42b92188c4ec51d2cd9a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3783475
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Gavin Mak d22bf60721 Add number of presubmit errors/warnings in header
Headers currently have the format: `** Presubmit ${name} **`. Make these
more helpful by including the number of total errors there are.

Bug: 1341987
Change-Id: Ib41a133c31568d4264d73c14350f0bbd9590356a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3756169
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Bruce Dawson 8254f06ba7 Mitigate Python 3 multiprocessing bug on Windows
The multiprocessing module on Windows has a bug where if you ask for
more than 60 child processes then it will hang. This is related to the
MAXIMUM_WAIT_OBJECTS (64) limit of WaitForMultipleObjects. Other sources
have listed the multiprocessing limit as being 61, or have said that the
the maximum number of objects that can be waited on is actually 63, but
those details don't really matter.

The original fix for this class of issues was crrev.com/c/2785844. This
change extends those fixes to depot_tools, which was missed last year.
This change also updates how PyLint is called by further limiting the
number of jobs to the number of files being processed divided by 10.
This is because there is a significant cost to creating PyLint
subprocesses - each takes about 0.5 s on my test machine. So there needs
to be enough parallelism to justify this.

Patches for PyLint and a bug for cpython are planned.

This will stop PyLint from hanging during presubmits on many-core
machines. The command used to reproduce the hangs and validate the fix
was:

  git cl presubmit -v --force --files "chrome/test/mini_installer/*.py"

Prior to this change this command would use (on my many-core test
machine) 96 processes and would hang. How it uses just two processes
because there are only 16 files to analyze.

Output before:
  Pylint (16 files using ['--disable=cyclic-import'] on 96 cores)
Output after:
  Pylint (16 files using ['--disable=cyclic-import'] on 2 processes)

This is actually not quite true because the hang would prevent the
old message from being displayed.

Bug: 1190269, 1336854
Change-Id: Ie82baf91df4364a92eb664a00cf9daf167e0a548
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3711282
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Bruce Dawson 10a8286824 Don't halt presubmits on exceptions
Some presubmit failures result in uncaught exceptions which historically
have halted presubmit runs. This behavior makes it more difficult to
find all failures, measure presubmit times, or detect other behavior
over time. This is particularly problematic when running "git cl
presubmit --all".

As an example, presubmit --all was broken and would not complete from
when crrev.com/3642633 landed until crrev.com/c/3657600. The minimal
repro was this presubmit command:

    git cl presubmit --files ui\accessibility\ax_mode.h

This command, or presubmit --all, triggered this error, which halts the
presubmits:

Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'
Traceback (most recent call last):
  File "presubmit_support.py", line 2038, in <module>
    sys.exit(main())
  File "presubmit_support.py", line 2015, in main
    return DoPresubmitChecks(
  File "presubmit_support.py", line 1738, in DoPresubmitChecks
    results += executer.ExecPresubmitScript(presubmit_script, filename)
  File "presubmit_support.py", line 1602, in ExecPresubmitScript
    self._run_check_function(function_name, context, sink,
  File "presubmit_support.py", line 1640, in _run_check_function
    six.reraise(e_type, e_value, e_tb)
  File "C:\Users\brucedawson\.vpython-root\e726d2\lib\site-packages\six.py", line 686, in reraise
    raise value
  File "presubmit_support.py", line 1630, in _run_check_function
    result = eval(function_name + '(*__args)', context)
  File "<string>", line 1, in <module>
  File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
    errs.extend(CheckModesMatch(input_api, output_api))
  File "PRESUBMIT.py", line 283, in CheckModesMatch
    ax_modes_in_js = GetAccessibilityModesFromFile(
  File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
    for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'

With this change the exception is handled and presubmits continue after
printing this message:

Evaluation of CheckChangeOnCommit failed: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js', Traceback (most recent call last):
  File "presubmit_support.py", line 1630, in _run_check_function
    result = eval(function_name + '(*__args)', context)
  File "<string>", line 1, in <module>
  File "PRESUBMIT.py", line 314, in CheckChangeOnCommit
    errs.extend(CheckModesMatch(input_api, output_api))
  File "PRESUBMIT.py", line 283, in CheckModesMatch
    ax_modes_in_js = GetAccessibilityModesFromFile(
  File "PRESUBMIT.py", line 245, in GetAccessibilityModesFromFile
    for line in open(fullpath).readlines():
FileNotFoundError: [Errno 2] No such file or directory: 'chrome/browser/resources/accessibility/accessibility.js'

That is, the crucial information regarding the failure is printed, but
subsequent presubmits are not skipped.

Bug: 1309977
Change-Id: I7cfeda85c830e6c8e567c0df3c50f27af1dbe835
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3653978
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Bruce Dawson 09c0c073ea Optimize presubmit --all with --no_diffs option
Some of the expensive checks when running presubmit --all, such as
CheckStableMojomChanges (~300 s) and CheckAddedDepsHaveTargetApprovals
(~200 s) only look at diffs and are therefore guaranteed to be NOPs when
running presubmit --all or --files=. Passing along the no_diffs state
lets these expensive checks be skipped, thus allowing for faster
iteration times.

Initial testing suggests that (with some supporting changes in the
Chromium repo) this reduces "presubmit --all" times by about ten
minutes, or a bit more than 10%, and additional improvements may be
possible.

Special handling for the no-diffs case also offers a simple way to avoid
presubmit failures that happen whenever all files are flagged as being
changed.

Finally, and perhaps most importantly for having a presubmit --all bot,
when --no_diffs is passed we can treat errors like "Issue wasn't
uploaded" and "Add a description to the CL" as messages, thus making it
possible to have zero presubmit errors when run on origin/main.

Bug: 1320937, 1322936
Change-Id: I0d09dd4aae8fdaa48c8b2f89337441cf96dcff72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3628368
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
3 years ago
Bruce Dawson 76fe1a701b Print presubmit messages in a consistent order
Presubmit message types were printed in an inconsistent order, generally
determined by which type was encountered first. This unpredictability
can be confusing when comparing runs. This change enforces a consistent
order, in increasing order of severity. ERRORS always go last so that
they will be the most visible when running presubmits locally.

This also adds a "There were Python %d presubmit errors." message to
the end to make it easier to tell when there were presubmit errors.
This is particularly useful when manually running presubmits.

Bug: 1309977
Change-Id: Ib2b73c7625789bad5b21ae12abf238b746cd11e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3668724
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Josip Sokcevic 632bbc0cb1 Skip Python 2 presubmit step when unneeded
All of the PRESUBMIT.py files in the Chromium repo are running under
Python 3. However "git cl presubmit" also works with other repos where
some PRESUBMIT.py scripts still run under Python 2. This means that
the Python 2 presubmit commit checks step cannot simply be disabled.
That meant that Chromium was paying up to a one-minute cost just to
setup for and look for Python 2 scripts that it doesn't run.

This change runs the Python 3 PRESUBMIT.py scripts first, and keeps
track of whether any were skipped. If none were skipped then the
Python 2 PRESUBMIT.py stage can be skipped.

Note that the child scripts of PRESUBMIT.py scripts may still be run
under Python 2, but that is orthogonal to this change.

Bug: 1313804
Change-Id: Ib65838223f232f1e78058d6a08ea15a89f442310
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3614453
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Bruce Dawson 31bfd51995 Actually add a trailing slash to dir_with_slash
AffectedFiles carefully added a slash to dir_with_slash, but then called
normpath - which trims trailing path separators. This meant that
ui/views/PRESUBMIT.py would analyze files in ui\views_content_client,
leading to spooky action at a distance. This was noticed when this
command triggered a presubmit error:

    git cl presubmit "--files=ui/views_content_client/*.cc;ui/views/readme.md"

when running presubmit on either file individually found nothing.

This change was tested by getting CheckChangeLintsClean to print the
files that it was asked to analyze, making the behavior quite obvious.

This bug appears to have existed for about thirteen years, but only
triggers in rare cases, and even then the incorrect behavior was almost
impossible to notice.

One of the tests was inadvertently testing the broken AffectedFiles
behavior and another seemed to be requiring it to handle '.'
correctly which I'm not sure we want to support.

Bug: 1309977
Change-Id: Ibdc39981d69664b03448acb228d4ab05b49436f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3632034
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
3 years ago
dpapad 443d9135cc Include TypeScript .ts files in DEFAULT_FILES_TO_CHECK.
These were previously erroneously skipped resulting in missing
violations.

Fixed: 1322170
Change-Id: I62d27f60bc1761f1f8809c6a7db2ca5f4e05e120
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3626289
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Demetrios Papadopoulos <dpapad@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Demetrios Papadopoulos <dpapad@chromium.org>
3 years ago
Bruce Dawson db8622bc6b Ensure _PresubmitResult._message is text
It is easy to get type confusion and end up passing a list as the
message parameter to _PresubmitResult. This error will not be detected
until the end of the run - perhaps hours later - when all evidence of
where the list came from is lost.

This change ensures that the message parameter is a string. If it is not
then the exception that is thrown should allow quick identification of
the problematic code.

This also fixes a presubmit unit test that passed None as the message.
We could support that but I don't think that we should.

Bug: 1309977
Change-Id: Ifb1d5100d47922b0ebd8bb834caa6fbba690b43c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3566436
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Josip Sokcevic 017544dc14 Add git cl presubmit --files
Chromium's presubmits contain a lot of latent errors - presubmit errors
that will trigger the next time a file is modified, but have been
waiting for years. Flushing those out with "git cl presubmit --all"
works poorly because it exercises all presubmits simultaneously, which
is too slow and triggers too many failures. Adding a --files option lets
small areas of the tree or specific file types to be exercised in a
controlled manner.

Bug: 1311697
Change-Id: I36ec6a759a80000d6ed4a8cc218ece327d45f8d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3559174
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Louis Romero 4ca9e1c783 Remove extraneous newlines from presubmit output
Prior to this CL, we have:

```
Running Python 2 presubmit upload checks ...

Python 2 presubmit checks passed.
Running Python 3 presubmit upload checks ...

Presubmit checks took 10.2s to calculate.

Python 3 presubmit checks passed.
 ios/chrome/browser/... | 20 +++++++++-----------
...
```

The first two checks are harder to parse with the newlines. This CL removes
them:

```
Running Python 2 presubmit upload checks ...
Python 2 presubmit checks passed.

Running Python 3 presubmit upload checks ...
Presubmit checks took 10.2s to calculate.
Python 3 presubmit checks passed.

 ios/chrome/browser/... | 20 +++++++++-----------
...
```

Change-Id: Iaf7ac4b203c401f60d74884fd9bdd3ba4ee12c20
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3512873
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Louis Romero <lpromero@chromium.org>
3 years ago
Josip Sokcevic e293d3dce9 Support py3 in post upload presubmit hooks
Currently, post upload presubmit hooks are exlusively executed with py2,
regardless of USE_PYTHON3 magic variable. This change adds py3 support
in the same fasion as regular presubmit hooks.

R=aravindvasudev@google.com

Fixed: 1297712
Change-Id: Ib464f8563e4135a63fc48692d27c8692fe1f630b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3469285
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan c5f0cbb865 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 42c5bbbc96 Revert "Use pylint 2.7 for depot_tools"
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan 22bf605bb6 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Takuto Ikuta 26bfecdd07 include filename in LUCI config validation
Fixed: 1254615
Change-Id: Ic6d5350c7a89a597c1126ec05235a3e63eb2ad09
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3312231
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Anthony Polito <apolito@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
3 years ago
Josip Sokcevic b6c1ed40d5 Make presubmit_unittest py3 compatible
And run it in presubmit checks

R=apolito@google.com

Change-Id: I1521fe4dd9fd5fe391459a00a91f03399b15b36a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3292211
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
3 years ago
Josip Sokcevic 6635baff5b Make CannedChecksUnittest py3 compatible
R=apolito@google.com

Change-Id: I24610ca05536b4ac6eed2c0a33ef411650b46dd0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3292089
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Josip Sokcevic 662d513faa Remove unused function GetPreferredTryMasters
Only usage was in v8, and that was removed in 2016:
http://crrev/45ec73da150e02fac1fc1dc15e3f14cb5bc2b45f

R=tandrii@chromium.org

Change-Id: I6aebcf1fc0cc29440b0b9c9b45d4e3fc35a38e53
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3214209
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
Josip Sokcevic f9d6b219ec Remove unused variables
R=apolito@google.com

Bug: 1098562
Change-Id: I98123cba5b6a2ff077a5a599c8640e18de3c4957
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3095027
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Anthony Polito <apolito@google.com>
Reviewed-by: Anthony Polito <apolito@google.com>
4 years ago
Josip Sokcevic 48c947534d Add presubmit notice if buganizer bug is malformed
If user enters buganizer bug prefixed with b/, warn user that b: is
expected.

R=apolito@google.com

Bug: 1226474
Change-Id: Ifbced843f9eecc00560cc273ae06bff9d435a815
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3016587
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
4 years ago
Peter Wen 866be0f290 Add an option to specify pylint version
This allows us to migrate PRESUBMIT.py files over to pylint-2.6 bit by
bit. It also allows us to support and upgrade pylint to newer versions
in the future.

Bug: 1221143
Fixed: 1221143
Change-Id: I9af96f5f06caf48e9923ad5fae75b98a0a7aeb49
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2994723
Commit-Queue: Peter Wen <wnwen@chromium.org>
Auto-Submit: Peter Wen <wnwen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
4 years ago
Dirk Pranke 61bf6e8d69 Add support for running PRESUBMIT.py checks under Python2 or 3.
This CL adds support for running PRESUBMIT.py under either Python2
or Python3 as specified in each PRESUBMIT.py file.

To run the checks under Python3, the PRESUBMIT.py file must contain
a line exactly matching "^USE_PYTHON3 = True$". If the file
does not contain this string, the checks will run under Python2.

Different PRESUBMIT.py files in a single CL may thus contain
a mix of 2- and 3-compatible checks, but each individual file will
only be run in one or the other (it doesn't likely make sense to run
them in both by default).

Bug: 1157663
Change-Id: Ic74977941a6519388089328b6e1dfba2e885924b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2832654
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
4 years ago
Stephen Martinis 7837307d38 Delete CheckBuildbotPendingBuilds
This check has been disabled basically everywhere since about 2013. It
should be safe to remove this; there will be warnings when trying to
skip it, but it shouldn't fail any builds.

Recipe-Nontrivial-Roll: build
Bug: 1194216
Change-Id: I84878f9e46543ae71f7e4e057b99d15a034e0d0d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2794799
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
4 years ago
Edward Lesmes 8170c29fd1 Reland "presubmit: Skip owners checks if code-owners plugin is enabled."
This is a reland of 2cf835a9ba

URL-encoded repo path, so chromium/src becomes chromium%2Fsrc
as Gerrit expects

Original change's description:
> presubmit: Skip owners checks if code-owners plugin is enabled.
>
> If code-owners plugin is enable for the repo, skip owners check on
> commit, and skip checking owners format, as that will be done by
> the plugin.
>
> Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I3038590f3a92cbf7b6dc0ba6eb47f72593a2ccf7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775840
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
4 years ago
Edward Lesmes dc11c6bd18 Reland "presubmit: Use code-owners plugin if available."
This is a reland of 36de4be91e
Should be fixed after cl/363445481

Original change's description:
> presubmit: Use code-owners plugin if available.
>
> Change-Id: I56734de985731d007360a4a4e65f2b16de28b69a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2753894
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Change-Id: Id7d5c6c3bd0a246e3400ca99584bb41a92853d16
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774218
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
4 years ago
Dirk Pranke e4bab45474 Reland "presubmit: Don't skip OWNERS check when Bot-Commit+1 is present."
This reverts commit e7c0581740.

Reason for revert: Whoops. chromium/src was updated. I was confused by a conversation on the #ops slack channel and my checkout being slightly out of date. I think the original change was actually okay.

Original change's description:
> Revert "presubmit: Don't skip OWNERS check when Bot-Commit+1 is present."
>
> This reverts commit 9757ad5883.
>
> Reason for revert: We need to update callers first (e.g., //PRESUBMIT.py in chromium/src).
>
> Original change's description:
> > presubmit: Don't skip OWNERS check when Bot-Commit+1 is present.
> >
> > Change-Id: I17b07796a86c5214e13a0058428889c1bb2b850d
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774080
> > Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> > Reviewed-by: Jason Clinton <jclinton@chromium.org>
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
>
> Change-Id: I02c3d5ea2e65ef852d34a6816d04fe1cad82823a
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775101
> Auto-Submit: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Change-Id: I2c1ae8c60938cbd9316e9075488cc7451068a2f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775103
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Auto-Submit: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
4 years ago
Dirk Pranke e7c0581740 Revert "presubmit: Don't skip OWNERS check when Bot-Commit+1 is present."
This reverts commit 9757ad5883.

Reason for revert: We need to update callers first (e.g., //PRESUBMIT.py in chromium/src).

Original change's description:
> presubmit: Don't skip OWNERS check when Bot-Commit+1 is present.
>
> Change-Id: I17b07796a86c5214e13a0058428889c1bb2b850d
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774080
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Jason Clinton <jclinton@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Change-Id: I02c3d5ea2e65ef852d34a6816d04fe1cad82823a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775101
Auto-Submit: Dirk Pranke <dpranke@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
4 years ago
Edward Lesmes 9757ad5883 presubmit: Don't skip OWNERS check when Bot-Commit+1 is present.
Change-Id: I17b07796a86c5214e13a0058428889c1bb2b850d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774080
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Edward Lesmes 48492322a0 Revert "presubmit: Use code-owners plugin if available."
This reverts commit 36de4be91e.

Reason for revert:
Not working for v8: crbug.com/1189233

Original change's description:
> presubmit: Use code-owners plugin if available.
>
> Change-Id: I56734de985731d007360a4a4e65f2b16de28b69a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2753894
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Bug: 1189233
Change-Id: Id96277e1a6d3b9dcf5fc1bf4816706515b75e55c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2767720
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
4 years ago