Commit Graph

446 Commits (ab918ca0bcae78caa8904d9224895ec86f151d49)

Author SHA1 Message Date
Anton Bershanskyi ab918ca0bc Reland "Add AffectedFile.Extension() in presubmit_support"
This is a reland of commit d447f41294

This reland excludes the change to PRESUBMIT.py which originally
caused regression and because of which the original commit was
reverted.

Original change's description:
> Add AffectedFile.Extension() in presubmit_support
>
> Add convenience method AffectedFile.Extension() for extracting file
> extension in a standard way. Add a test for this new functionality.
> Add a similar test for AffectedFile.UnixLocalPath().
>
> Bug: None
> Change-Id: If591e751fb2e4fb5355ad0b6f93f310667849d68
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6273341
> Commit-Queue: Anton Bershanskyi <bershanskyi@gmail.com>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>

Bug: None
Change-Id: Ie5fc551b9c0818dfd9a1c571cc247cc157dbc8b1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6335512
Commit-Queue: Anton Bershanskyi <bershanskyi@gmail.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
2 months ago
Gavin Mak 9f715f8017 Revert "Add AffectedFile.Extension() in presubmit_support"
This reverts commit d447f41294.

Reason for revert: "AttributeError: 'str' object has no attribute 'Extension'"

Original change's description:
> Add AffectedFile.Extension() in presubmit_support
>
> Add convenience method AffectedFile.Extension() for extracting file
> extension in a standard way. Add a test for this new functionality.
> Add a similar test for AffectedFile.UnixLocalPath().
>
> Bug: None
> Change-Id: If591e751fb2e4fb5355ad0b6f93f310667849d68
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6273341
> Commit-Queue: Anton Bershanskyi <bershanskyi@gmail.com>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Gavin Mak <gavinmak@google.com>

Bug: None
Change-Id: Ic51b0942e3a3267194fb73af5e92c902b2098c4c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6279837
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 months ago
Anton Bershanskyi d447f41294 Add AffectedFile.Extension() in presubmit_support
Add convenience method AffectedFile.Extension() for extracting file
extension in a standard way. Add a test for this new functionality.
Add a similar test for AffectedFile.UnixLocalPath().

Bug: None
Change-Id: If591e751fb2e4fb5355ad0b6f93f310667849d68
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6273341
Commit-Queue: Anton Bershanskyi <bershanskyi@gmail.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 months ago
Dirk Pranke 7cffe56af4 Add UnixLocalPath*() in presubmit_support.
This adds AffectedFile.UnixLocalPath(), Change.UnixLocalPaths(), and
input_api.UnixLocalPaths() to the presubmit support APIs, so that we can
check paths in a platform-independent manner (i.e., not have to worry
about whether we're running on Windows and might need to handle
backslashes).

Change-Id: I1a424825384cb950ef2d4d3008c9c6e60028cf07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6236404
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
3 months ago
Gavin Mak 77a800d729 Print branch name when running presubmits
This implements the proposed solution B in https://crbug.com/380877175#comment5
and will help distinguish presubmit results when working with stacked changes.

Bug: 380877175
Change-Id: I7f0807f7ce03109288976069c57c28488d6cadfd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6196130
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 months ago
Mike Frysinger 7f4762b2a1 pylintrc: version all of them
When running pylint tools in depot_tools itself, we default to the
pylintrc file all the time even though we have versioned ones we
want to use instead.  This breaks things like `./pylint-2.17 foo.py`.
Rename the existing pylintrc to 2.6 and duplicate it to 2.7 so we
clear out the 'pylintrc' file and the wrappers always default to the
versioned ones.

We've been riding on pylint being a bit compatible across versions,
but it seems to be accelerating in adding/removing diagnostics and
settings, so this is more churn to add new versions, but we don't
have much choice.

Change-Id: I5c20a41ed8aead3e90b6dda8f5f2fc3b87cfb251
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/6150453
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
4 months ago
Gavin Mak a6ce588715 Use ProvidedDiffChange if diff is empty string
A ProvidedDiffChange should be able to use an diff, even if it is empty.

Bug: 372936485
Change-Id: If6bea0063e92c8942aed080e4c26b8d032b90faa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5926311
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Scott Lee <ddoman@chromium.org>
6 months ago
Bruce Dawson 9157dd872e Remove extraneous Python version from presubmit warnings
When a presubmit warning is triggered this message is printed:

    There were Python 3 presubmit warnings.

The Python version information was useful when we were transitioning
our presubmits from Python 2 to Python 3, but that transition is long
finished, and now that information is just noise.

This change deletes the excess information.

Change-Id: I4e43e1c9daf36daaaa9430d76ad557c800cf962b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5907924
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
7 months ago
Joanna Wang d43a3eb748 Add end_commit for calculating GitChange diffs.
Bug: b/365162784

Change-Id: I97b69b1ae5183bbdaa7d4de0b1f23e62f4e1d8d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5853908
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
8 months ago
Gavin Mak 37a737c1bc Update presubmit feedback message for Cog
We shouldn't be suggesting people run unsupported commands on cog.

Change-Id: I0a15b825b32a3f9bc40c65e0b51be33f06de54ad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5782965
Auto-Submit: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
9 months ago
Gavin Mak 8ac3425344 Create scm.DIFF.GetAllFiles
For a ProvidedDiffChange, the AllFiles method naively returns all files
with rglob("*"). The returned list includes files in nested submodules.
This does not match the behavior of GitChange's AllFiles which uses
git ls-files to find all files.

Implement a new SCM that stops iterating recursively when it
sees a submodule from .gitmodules in the repo root.

Bug: b/323243527
Change-Id: I170d0f1bc4a838acea04779dee3df7fca0bce359
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5648616
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
10 months ago
Gavin Mak 14cccc4b9a Make CheckChangedLUCIConfigs work on non-git workspace
The check relies on cl.GetRemoteBranch() and cl.GetRemoteUrl() that
both rely on git. Provide that info with Gerrit info from input_api
if available instead of always calling git.

Bug: 333744051
Change-Id: I915d6451d808c9bf871804d116ea884294ee7c84
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5479889
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Riley Wong <rgw@google.com>
1 year ago
Gavin Mak 715d94600f Add --generate_diff flag to presubmit_support.py
This flag allows a diff to be generated based on an upstream commit
for provided files and for checks to be run with on that diff.

Bug: b/323243527
Change-Id: Ia3f4ec50b15eabb90d391b9bc795fc4b35a35a08
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376402
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
1 year ago
Gavin Mak 62dc9f462e Implement support for change info thru diff file
This will be used in cases where git workspace is unavailable.

Bug: b/323243527
Change-Id: I51056cfbac50f76a0de1b98cfec488d8f133e353
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5263021
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Gavin Mak b1b1a43f07 Move LSC check into pan project presubmit checks and check num files
git cl upload prompts a user if they upload a particularly large change,
determined by the number of CCs added on the change. With the watchlists
analyzer and eventual removal of CCing watchers from git cl, this check
needs to be updated to work without knowing the number of CCs.

This change does two things:
1. Moves the check from git cl upload to a pan-project presubmit check.
2. Update the heuristic of a LSC from > 100 CCs to > 100 files modified.

Bug: 324562917
Change-Id: I6bd0db9fde942273acebc320c8b52e81b02bd42f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5296199
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
1 year ago
Gavin Mak 4c331a1ee2 Reland "Reland "Refactor git functionality out of Change and _DiffCache""
This is a reland of commit 4d2864f3a1

The previous change initiated "self._diff_cache()" per file.

Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Bug: b/323243527
Change-Id: I4733b6870935d4200cb32255214c57cdb55e84f0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5286636
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Arthur Sonzogni aa475e6a12 Revert "Reland "Refactor git functionality out of Change and _DiffCache""
This reverts commit 4d2864f3a1.

Reason for revert: This still doesn't scale well with large commit:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8756578662173359185/+/u/presubmit/stdout?format=raw

This is now a different instruction called repeatedly needlessly:
```
git -c core.quotePath=false diff -p --no-color --no-prefix --no-ext-diff f0b151b0e519ceb3efae2ad8951d8a9ce86e4fbd...HEAD --no-renames;  cwd=D:\b\s\w\ir\cache\builder\win_presubmit\src
```
called for each affected files.

We probably want to cache the output for this too?

Original change's description:
> Reland "Refactor git functionality out of Change and _DiffCache"
>
> This is a reland of commit dd1a596c3e
>
> It is not a pure reland and adds support for caching submodules.
>
> Original change's description:
> > Refactor git functionality out of Change and _DiffCache
> >
> > A followup change will add support for change diff provided as user
> > input through stdin/file.
> >
> > Bug: b/323243527
> > Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
> > Commit-Queue: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
>
> Bug: b/323243527
> Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>

Bug: b/323243527, b/324293047
Change-Id: Iffde46facefa9d2e3a0d3a9ab5f8e753c53ea6d4
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5272317
Auto-Submit: Arthur Sonzogni <arthursonzogni@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
1 year ago
Gavin Mak 4d2864f3a1 Reland "Refactor git functionality out of Change and _DiffCache"
This is a reland of commit dd1a596c3e

It is not a pure reland and adds support for caching submodules.

Original change's description:
> Refactor git functionality out of Change and _DiffCache
>
> A followup change will add support for change diff provided as user
> input through stdin/file.
>
> Bug: b/323243527
> Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Bug: b/323243527
Change-Id: I74bbb179d4cbda7431101a2d707131fab2093029
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5280620
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
1 year ago
Arthur Sonzogni b1d51e6113 Revert "Refactor git functionality out of Change and _DiffCache"
This reverts commit dd1a596c3e.

Reason for revert: This doesn't scale well with large commit:
https://issues.chromium.org/issues/324293047

Original change's description:
> Refactor git functionality out of Change and _DiffCache
>
> A followup change will add support for change diff provided as user
> input through stdin/file.
>
> Bug: b/323243527
> Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Bug: b/323243527
Change-Id: Ic5179de6744a2ead629f512a5098f6e8c63d01c8
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5277255
Commit-Queue: Gavin Mak <gavinmak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Arthur Sonzogni <arthursonzogni@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
1 year ago
Gavin Mak dd1a596c3e Refactor git functionality out of Change and _DiffCache
A followup change will add support for change diff provided as user
input through stdin/file.

Bug: b/323243527
Change-Id: I8d3420370e134859c61e35e23d76803227e4a506
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254364
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak 120efcb475 Remove _MergeMasters
This was added in https://codereview.chromium.org/178223016 and used
for rietveld which is no longer used.

Change-Id: Idbfdb50a61dba884f5212c1de904f83324a449f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5254600
Auto-Submit: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
1 year ago
Joanna Wang 46cb7d0aca Add AffectedSubmodules() to presubmit change.
+ Change scm GetAllFiles back to returning all files
  + should be safe to do since it's only used by presubmit code
+  add ignore_submodules option to CaptureStatus
  + It's used by other parts of depot_tools code
+ Add AffectedSubmodules to return submodules only
+ AffectedFiles still only returns non-submodule files.

Bug: 1475770
Change-Id: I457ad96099fb20eff4cc1a4fc84f59e7ae707abe
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5216005
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Yuanjun Huang 5dffbd43c4 [config] change to use luci-config v2 in presubmit
In `CheckChangedLUCIConfigs`, it will call luci-config to list all
config sets. Change that call to v2.

Change-Id: I7a00d157fa631174b2124f0920026fa6b1fea04c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4900076
Commit-Queue: Yuanjun Huang <yuanjunh@google.com>
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
2 years ago
Maggie Chen 34e0ecf20e Change from PresubmitError to PresubmitWarning for bad license
We cannot distinguish if this is a moved third-party file. So just
do warnings for a license header from a third-party company.

Bug: 1462922
Change-Id: I78394dcccb9028bce6c535dfce625364dd29157b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4895337
Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Maggie Chen <magchen@chromium.org>
2 years ago
Gavin Mak f936d540e1 Remove __future__ imports
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py

Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Yiwei Zhang 7a69b031d5 presubmit: support checking new TODO format
R=sokcevic

Bug: 1479023
Change-Id: I6eabb16447526fbc8de83b823763888aff1a8249
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4847314
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Yiwei Zhang <yiwzhang@google.com>
2 years ago
Mike Frysinger 677616322a tests: switch to 4 space indent
Reformat this dir by itself to help merging with conflicts with other CLs.

Reformatted using:
parallel ./yapf -i -- tests/*.py
~/chromiumos/chromite/contrib/reflow_overlong_comments tests/*.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
tests/bot_update_coverage_test.py
tests/cipd_bootstrap_test.py
tests/gclient_eval_unittest.py
tests/gclient_git_smoketest.py
tests/gclient_scm_test.py
tests/gclient_smoketest.py
tests/gclient_test.py
tests/gclient_transitions_smoketest.py
tests/gclient_utils_test.py
tests/git_cl_test.py
tests/git_hyper_blame_test.py
tests/git_rebase_update_test.py
tests/lockfile_test.py
tests/metrics_test.py
tests/presubmit_canned_checks_test.py
tests/presubmit_unittest.py
tests/roll_dep_test.py

Change-Id: I8fed04b4ba81d54b8f45da612213aad27a9e1a2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4842592
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
2 years ago
Yiwei Zhang fbb06cdb23 config: output warnings for validation error/warnings in non-affected files
R=iannucci, sokcevic, yuanjunh

Bug: 1449933
Change-Id: I863c55f5a12da103b4167c8cf9a1aac02962838f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4602126
Reviewed-by: Yuanjun Huang <yuanjunh@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Yiwei Zhang 422f4d59a4 config: use lucicfg validate to validate the config files
The new flow is that if any affected file is in the config directory
of a config set, the script will call `lucicfg validate` to validate
the whole config directory and then filter out the validation result
that does not belong to affected file(s). This will introduce a bit
more latency in valdiation because before, only affected files are
sent to LUCI Config for validation. However, this is not latency
senstively operation and always validating the full snapshot will
faciliate us breaking down the config as it's possible in order to
validate the affected file, LUCI Config will need an unaffected file.

R=iannucci, yuanjunh

Bug: 1449933
Change-Id: Id330973f59b882569a8ece36edb4b6a8ff0ed2d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4591233
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Yuanjun Huang <yuanjunh@google.com>
2 years ago
Bruce Dawson b6cb9e0b9a Remove Python 2 support for presubmit Commands
PRESUBMIT.py files can invoke other scripts, and those scripts may be
run under Python 2 or Python 3. There are multiple mechanisms that
govern which version will be used, sometimes requiring several flags
to be passed.

This change removes support for Python 2 from this system, thus making
it simpler to invoke Python 3 scripts.

The function parameters that are used to select Python versions are
passed in from multiple places so they still need to be supported, but
they are now ignored. The parameters are deleted to prevent accidental
use.

This change was tested by running this command in a Chromium repo:

  git cl presubmit --force --all

Bug: 1207012
Change-Id: I4adc7164417e155ff80d3a039cf36ed030756456
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4328470
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Josip Sokcevic a4b871ebd9 Disable showing callstacks when running async jobs
When using async jobs, information about presubmit is not present in
callstack and it's just noise.

Change-Id: Ib95af92f9e83f1fb8b8d68c7aece6bbb4b03e61c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4505697
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Daniel Cheng 541638fd23 Fix the duplicate CC issue, hopefully once and for all.
Change-Id: Idd08d28ddeab8b66df1ae9a51d581ceaf62ec984
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4522489
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
2 years ago
Bruce Dawson 4082f883ab Remove some Python 2 test code
Now that the presubmit system no longer supports Python 2 we can remove
support for it from the presubmit system tests. This change removes all
references to sys.version_info from this file.

This also fixes an unclosed file warning that suddenly appeared and
was stopping this change from landing.

Bug: 1207012
Change-Id: Ib74e9009b9d5cea0caf37c88e3258e9dd818f269
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4522416
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Robert Iannucci 3d6d2d2500 [git-cl][presubmit] Remove references to LUCI_OMIT_PYTHON2.
This changes all references to the LUCI_OMIT_PYTHON2 environment
variable to instead assume that it is `'true'`.

R=sokcevic, tikuta

Bug: 1441784
Recipe-Nontrivial-Roll: build
Change-Id: Iea161a372adb68cdcb330d131df2c19ca2fe7b37
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4522480
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 967cf672eb Revert "Reland "Remove Python 2 support for PRESUBMIT.py""
This reverts commit 119cff3d2f.

Reason for revert: some builders still call py2 presubmit
http://go/bbid/8781466817788761185

We need to clean up recipe first before relanding this

Original change's description:
> Reland "Remove Python 2 support for PRESUBMIT.py"
>
> This reverts commit d4c6cbeb61.
>
> Reason for revert: Python 2 invocations of the presubmit system
> should be gone now.
>
> Original change's description:
> > Revert "Remove Python 2 support for PRESUBMIT.py"
> >
> > This reverts commit 8454fc2458.
> >
> > Reason for revert: post submit hooks failing
> > https://crbug.com/1422416
> >
> > Original change's description:
> > > Remove Python 2 support for PRESUBMIT.py
> > >
> > > The presubmit system still supports invoking PRESUBMIT.py files using
> > > Python 2. This has recently been turned off on the bots so this change
> > > removes support more completely.
> > >
> > > There are still some python3 parameters being passed around - it seemed
> > > better to do the simplest possible removal now, with a follow-up change
> > > to remove more support code after this has sat for a while.
> > >
> > > Tests run from PRESUBMIT.py files could still be run using Python 2, but
> > > those should also have been addressed already. Removing support for that
> > > will be done in a subsequent change.
> > >
> > > Bug: 1207012
> > > Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
> > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
> > > Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> > > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> >
> > Bug: 1207012
> > Bug: 1422416
> > Change-Id: Iaf3102e63ec3c698d0258fac5746dbd92c30edbb
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4317176
> > Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
> > Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> > Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
>
> Bug: 1207012
> Bug: 1422416
> Change-Id: I9521095989c708188fca0251fa13881e086b1395
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4516015
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Bug: 1207012
Bug: 1422416
Change-Id: Ifa775ca19f8c7b3c166eb89a255c25c20aaf4a98
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4521578
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Bruce Dawson 119cff3d2f Reland "Remove Python 2 support for PRESUBMIT.py"
This reverts commit d4c6cbeb61.

Reason for revert: Python 2 invocations of the presubmit system
should be gone now.

Original change's description:
> Revert "Remove Python 2 support for PRESUBMIT.py"
>
> This reverts commit 8454fc2458.
>
> Reason for revert: post submit hooks failing
> https://crbug.com/1422416
>
> Original change's description:
> > Remove Python 2 support for PRESUBMIT.py
> >
> > The presubmit system still supports invoking PRESUBMIT.py files using
> > Python 2. This has recently been turned off on the bots so this change
> > removes support more completely.
> >
> > There are still some python3 parameters being passed around - it seemed
> > better to do the simplest possible removal now, with a follow-up change
> > to remove more support code after this has sat for a while.
> >
> > Tests run from PRESUBMIT.py files could still be run using Python 2, but
> > those should also have been addressed already. Removing support for that
> > will be done in a subsequent change.
> >
> > Bug: 1207012
> > Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
> > Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
>
> Bug: 1207012
> Bug: 1422416
> Change-Id: Iaf3102e63ec3c698d0258fac5746dbd92c30edbb
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4317176
> Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>

Bug: 1207012
Bug: 1422416
Change-Id: I9521095989c708188fca0251fa13881e086b1395
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4516015
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic a9a7eecf37 Revert "Extract searching relevat files in utils function"
This reverts commit 512fd3bd85.

Reason for revert: breaks some presubmits
https://crbug.com/1421441

Original change's description:
> Extract searching relevat files in utils function
>
> This change introduces lib/ directory which will be used for classes
> that are considered libraries and used by multiple executables.
>
> Change-Id: I3da126778bf5ffdb28d7a3c2b966c85de08ba717
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289679
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

Bug: 1421441
Change-Id: I20e40f452a544a1188a258131a620e2afcd64f13
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4328468
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 48d8e90bd4 Revert "Reland "Remove Python 2 support for PRESUBMIT.py""
This reverts commit e9ece0f581.

Reason for revert: broke infra presubmits:
https://ci.chromium.org/p/infra/builders/try/infra-try-presubmit

Original change's description:
> Reland "Remove Python 2 support for PRESUBMIT.py"
>
> This is a reland of commit 8454fc2458
>
> Original change's description:
> > Remove Python 2 support for PRESUBMIT.py
> >
> > The presubmit system still supports invoking PRESUBMIT.py files using
> > Python 2. This has recently been turned off on the bots so this change
> > removes support more completely.
> >
> > There are still some python3 parameters being passed around - it seemed
> > better to do the simplest possible removal now, with a follow-up change
> > to remove more support code after this has sat for a while.
> >
> > Tests run from PRESUBMIT.py files could still be run using Python 2, but
> > those should also have been addressed already. Removing support for that
> > will be done in a subsequent change.
> >
> > Bug: 1207012
> > Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
> > Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
>
> Bug: 1207012
> Change-Id: If542cac21d8ec8704b28d03fd8407c5c2899ca2c
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4317177
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
> Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>

Bug: 1207012
Change-Id: Iebf76d9e2580761fc773791bac07439160503c98
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4323198
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Josip Sokcevic e9ece0f581 Reland "Remove Python 2 support for PRESUBMIT.py"
This is a reland of commit 8454fc2458

Original change's description:
> Remove Python 2 support for PRESUBMIT.py
>
> The presubmit system still supports invoking PRESUBMIT.py files using
> Python 2. This has recently been turned off on the bots so this change
> removes support more completely.
>
> There are still some python3 parameters being passed around - it seemed
> better to do the simplest possible removal now, with a follow-up change
> to remove more support code after this has sat for a while.
>
> Tests run from PRESUBMIT.py files could still be run using Python 2, but
> those should also have been addressed already. Removing support for that
> will be done in a subsequent change.
>
> Bug: 1207012
> Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Bug: 1207012
Change-Id: If542cac21d8ec8704b28d03fd8407c5c2899ca2c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4317177
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic d4c6cbeb61 Revert "Remove Python 2 support for PRESUBMIT.py"
This reverts commit 8454fc2458.

Reason for revert: post submit hooks failing
https://crbug.com/1422416

Original change's description:
> Remove Python 2 support for PRESUBMIT.py
>
> The presubmit system still supports invoking PRESUBMIT.py files using
> Python 2. This has recently been turned off on the bots so this change
> removes support more completely.
>
> There are still some python3 parameters being passed around - it seemed
> better to do the simplest possible removal now, with a follow-up change
> to remove more support code after this has sat for a while.
>
> Tests run from PRESUBMIT.py files could still be run using Python 2, but
> those should also have been addressed already. Removing support for that
> will be done in a subsequent change.
>
> Bug: 1207012
> Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
> Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Bug: 1207012
Bug: 1422416
Change-Id: Iaf3102e63ec3c698d0258fac5746dbd92c30edbb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4317176
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Bruce Dawson 8454fc2458 Remove Python 2 support for PRESUBMIT.py
The presubmit system still supports invoking PRESUBMIT.py files using
Python 2. This has recently been turned off on the bots so this change
removes support more completely.

There are still some python3 parameters being passed around - it seemed
better to do the simplest possible removal now, with a follow-up change
to remove more support code after this has sat for a while.

Tests run from PRESUBMIT.py files could still be run using Python 2, but
those should also have been addressed already. Removing support for that
will be done in a subsequent change.

Bug: 1207012
Change-Id: Id244d547a04438f83734dba269c3cc180c148b37
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4315183
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Josip Sokcevic 7958e30579 Revert "Move scm.py to lib/"
This reverts commit 688adfe276.

Reason for revert: breaks try jobs

Original change's description:
> Move scm.py to lib/
>
> It belongs there.
>
> Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

Change-Id: I662907b8bb4b2e7db82409bd34c3905db7961a12
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4300261
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
2 years ago
Josip Sokcevic 688adfe276 Move scm.py to lib/
It belongs there.

Change-Id: I286b50c43c587c39557fc508175efbd4b0f51d51
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289680
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Josip Sokcevic 512fd3bd85 Extract searching relevat files in utils function
This change introduces lib/ directory which will be used for classes
that are considered libraries and used by multiple executables.

Change-Id: I3da126778bf5ffdb28d7a3c2b966c85de08ba717
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4289679
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Brian Ryner 790a0c522d Fix CheckCIPDManifest for python 3.
Change-Id: Icf7b8e7ad732a731520550ff9e276f13016effb7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4149964
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Brian Ryner <bryner@google.com>
2 years ago
Pavol Marko 624e7eec34 Change working directory to PRESUBMIT's dir also for postsubmit
While we're at it, ensure that the working directory is reverted even if
exceptions are raised by moving that part into a finally block.

Bug: 1404222
Change-Id: I8606bc5810202ff1ecdc7f202e3f574bf643f91d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4129515
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Pavol Marko <pmarko@chromium.org>
2 years ago
Bruce Dawson 7a81ebf111 Add .rs to list of presubmit source file patterns
.rs files were not part of DEFAULT_FILES_TO_CHECK which means that they
were not being checked for proper licenses, among other things.

Testing with this command reveals that six missing or bad licenses have
crept in:

    git cl presubmit --force --files "*.rs"

Output was:

Found a bad license header in these files:
  build\rust\tests\test_aliased_deps\main.rs
  build\rust\tests\test_aliased_deps\real_name.rs
  mojo\public\rust\tests\util\mojom_validation.rs
  testing\rust_gtest_interop\gtest_attribute.rs
  tools\crates\gnrt\sample_package\foo\src\lib.rs
  tools\crates\gnrt\sample_package\src\main.rs

Change-Id: Ia7788d92d428a4060f4a0b854fdcee0450b0a3f6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4128762
Commit-Queue: Gavin Mak <gavinmak@google.com>
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Collin Baker <collinbaker@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
2 years ago
Bruce Dawson 31ca877242 Fix CheckLicense tests
A last minute change to the CheckLicense tests in crrev.com/c/4032366
used the _GetLicenseText function in the wrong place, thus neutering one
test and duplicating code in another. This fixes the tests so that they
minimize duplication and actually test the CSS/C style comments.

Change-Id: Ibbe880dec8d1e85416ed1b7ce94922c6f2c2a543
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4037103
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2 years ago
Bruce Dawson 25cf78395c Fix two CheckLicense bugs and improve tests
Some recent "improvements" to CheckLicense were buggy, at least
partially because I did not increase test coverage when adding new code
paths. This change fixes two known mistakes and greatly increases test
coverage.

One bug was that the code assumed that match.groups()[0] would always be
the current year, but that assumption is invalid when a license regex is
passed in.

Another bug is that the new_license_re took away the possibility of the
license ending with */ but that is used in .css files.

The increased test coverage catches these and validates other
assumptions around new files.

The failures were reproduced and then the fixes validated using this
command:

  vpython3 tests/presubmit_unittest.py

Bug: 1098010
Change-Id: Ic91ca9f7a182660aef7b1eead79e841f58e15a3e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4032366
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
2 years ago