Chromium's presubmits contain a lot of latent errors - presubmit errors
that will trigger the next time a file is modified, but have been
waiting for years. Flushing those out with "git cl presubmit --all"
works poorly because it exercises all presubmits simultaneously, which
is too slow and triggers too many failures. Adding a --files option lets
small areas of the tree or specific file types to be exercised in a
controlled manner.
Bug: 1311697
Change-Id: I36ec6a759a80000d6ed4a8cc218ece327d45f8d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3559174
Auto-Submit: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
When running git cl presubmit --all which is handy for finding
latent presubmit bugs, the 'No diff found for %s' gets printed many
thousands of times, making actual issues difficult to find. This change
suppresses that message, to make actual issues easier to find.
The message for slow presubmits prints the name of the slow presubmit,
but there are 243 different CheckChangeOnCommit functions, so the name
is actually not sufficient. Therefore this change plumbs through the
path to the script containing the presubmit and prints it.
Similarly, the cpplint message "Done processing %s" doesn't add enough
value.
These changes will make it easier to find the signal in the presubmit
noise.
Bug: 1309977
Change-Id: Iba40b5748266e3296eeb530bb00182db4814aa5d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3556594
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
depot_tools has no versioning. It's hard to know if reported issues were
caused due to outdated depot_tools or actual unresolved bug.
This CL adds basic information about depot_tools version and it's
included in presubmit failure.
R=aravindvasudev@google.com, gavinmak@google.com
Change-Id: If8577c0826063693a7278a57a0cce629d4b1325f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3541061
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
git cl presubmit has --all option which marks all files as modified.
However, that doesn't work well with SCM diff. That is, git diff may not
contain affected file. Instead of throwing an exception in such case,
just return no diff.
R=gavinmak@google.com
Fixed: 808346
Change-Id: Ie1d534b8eebc84bc5206eba7db8057829390fbec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3485501
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Currently, post upload presubmit hooks are exlusively executed with py2,
regardless of USE_PYTHON3 magic variable. This change adds py3 support
in the same fasion as regular presubmit hooks.
R=aravindvasudev@google.com
Fixed: 1297712
Change-Id: Ib464f8563e4135a63fc48692d27c8692fe1f630b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3469285
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This reverts commit 22bf605bb6.
Reason for revert: breaks gclient sync
Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.
Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
Gitwatch supports closing bugs with a FIX(ES|ED|ING)? tag in the
commit message, but we only check BUG to ensure a bug is
specified, add the fix fields to BugsFromDescription so that presubmits
will still pass if only eg: FIXES=b:1234 is given.
Gitwatch reference:
http://shortn/_LGPgBjgi3A
FIXED=b:203812728
TEST=presubmit_unittest.py doesn't fail more than it already does
Change-Id: I6afc38c786e281dcefb4d359bb9212ebe5980ed6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3237598
Auto-Submit: Sean McAllister <smcallis@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
When presubmit checks fail, we want to print the error message from the
underlying check/tool. We use _PresubmitResult objects to collect these
diagnostics and output them to the console and json files.
This change ensures that the string values passed to the
_PresubmitResult constructor will be converted, as needed, to 'str'
instances. This way, we will avoid encoding errors and make sure to
print the actual messages instead of repr(message).
SHERIFFS: this patch breaks backwards compatibility of presubmit error
handling but we don't expect issues in downstream projects. Please
revert if downstream presubmit checks start failing spuriously.
Bug: 1225052
Change-Id: If7f5c99cb5d730c1bfbd6d108b912f77b5f2455c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3037262
Commit-Queue: Tom McKee <tommckee@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This CL adds a new field to the codereview.settings file used by
git_cl for project-wide defaults. If `USE_PYTHON3` is set to True,
then we will run the PRESUBMIT checks under Python3 by default
instead of Python2, unless the PRESUBMIT.py file contains
`USE_PYTHON3 = False` on a line by itself
(as opposed to now, where we'll use Python2 by default unless the
file contains `USE_PYTHON3 = True`).
This will allow us have Python3 be the default for new files
and to eventually eliminate any uses of `USE_PYTHON3` from the
individual presubmit files. Of course, you will have to go in and
explicitly add `USE_PYTHON3 = False` to any Py2-only files prior
to enabling this.
Bug: 1207012
Change-Id: Id8ec45b43940e5bcffeee196398c711c541e733e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2917747
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Currently, GitChange.AllFiles returns a list of byte[]. This is
inconsistent with py2 and may cause issues when byte[] is used when
string is expected.
Change-Id: I3e73bc5f71aa787f55357b8139cdeeabee056047
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2946934
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
When presubmit checks create warnings that should raise a prompt
whether or not to continue, the output wasn't being flushed
properly under python3.
Bug: 1213316
Change-Id: Ibb4b4d965d49f0cd65ccc0737282e5f5b5f58004
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2921323
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Commit 6fc394f added some diagnostics, but it didn't quite work out the
way it was planned.
Bug: 1210746
Change-Id: Ia4f30606e87d3f8b1362441bd74bb7ed0f5ca886
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2928771
Auto-Submit: Greg Thompson <grt@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
If a presubmit happened to generate a crash, the logic we have
for reporting the presubmit check times would call `six.reraise()`
incorrectly, leading to a second crash.
This CL fixes the invocation of the second crash, and also adds
an error message to help users tripping over one possible source
of the first crash (reading a binary file as text).
Bug: 1210746
Change-Id: Ic46f38901b6acf2055b3feb7272dc751dc69037c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2921322
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
Presubmits when uploading occasionally take a very long time - over a
minute - and after the fact there is no easy way to know why. This makes
fixes to slow presubmits take longer.
This change builds on crrev.com/c/2532895 to print a message when any
individual presubmit function takes longer than ten seconds. During
normal usage this is a NOP but it will presumably find the long poles
when presubmits are running particularly slowly.
This is similar to gclient runhooks where any hooks that take longer
than ten seconds have their run time printed.
Change-Id: If57ed35d7a7d221f6380e9b97cf72af56f75e441
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2911594
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
CheckForWindowsLineEndings is a Chromium presubmit to make sure that
\r\n line endings (Windows line endings) don't get committed. Among
other things these line endings mess up the license checks, leading to
cryptic errors.
The problem is that ChangedContents() was stripping line endings, making
any checking for them futile. This change adds an option to preserve
line endings, to be used by Chromium's CheckForWindowsLineEndings.
It's not clear how long this has been broken.
See also crrev.com/c/2911914 which must land after this.
Bug: 801033
Change-Id: I7cdb9b3581788624f9eca081da43bb070ee412a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2910488
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
This fixes CL:2880843 to use a module already present in vpython.
Bug: 1206782
Change-Id: If4158a62011e043e89f40f368c9c644434df6614
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2901758
Commit-Queue: Allen Webb <allenwebb@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This is a followup to CL:2797935 to make it easier to debug exceptions.
Bug: 1206782
Change-Id: Ibb649e94fd562919737de152e3203ca6b02b38ad
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2880843
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>
This CL adds support for running PRESUBMIT.py under either Python2
or Python3 as specified in each PRESUBMIT.py file.
To run the checks under Python3, the PRESUBMIT.py file must contain
a line exactly matching "^USE_PYTHON3 = True$". If the file
does not contain this string, the checks will run under Python2.
Different PRESUBMIT.py files in a single CL may thus contain
a mix of 2- and 3-compatible checks, but each individual file will
only be run in one or the other (it doesn't likely make sense to run
them in both by default).
Bug: 1157663
Change-Id: Ic74977941a6519388089328b6e1dfba2e885924b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2832654
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
This is a reland of 2cf835a9ba
URL-encoded repo path, so chromium/src becomes chromium%2Fsrc
as Gerrit expects
Original change's description:
> presubmit: Skip owners checks if code-owners plugin is enabled.
>
> If code-owners plugin is enable for the repo, skip owners check on
> commit, and skip checking owners format, as that will be done by
> the plugin.
>
> Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: I3038590f3a92cbf7b6dc0ba6eb47f72593a2ccf7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775840
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This is a reland of 36de4be91e
Should be fixed after cl/363445481
Original change's description:
> presubmit: Use code-owners plugin if available.
>
> Change-Id: I56734de985731d007360a4a4e65f2b16de28b69a
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2753894
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Change-Id: Id7d5c6c3bd0a246e3400ca99584bb41a92853d16
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774218
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit e7c0581740.
Reason for revert: Whoops. chromium/src was updated. I was confused by a conversation on the #ops slack channel and my checkout being slightly out of date. I think the original change was actually okay.
Original change's description:
> Revert "presubmit: Don't skip OWNERS check when Bot-Commit+1 is present."
>
> This reverts commit 9757ad5883.
>
> Reason for revert: We need to update callers first (e.g., //PRESUBMIT.py in chromium/src).
>
> Original change's description:
> > presubmit: Don't skip OWNERS check when Bot-Commit+1 is present.
> >
> > Change-Id: I17b07796a86c5214e13a0058428889c1bb2b850d
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2774080
> > Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> > Reviewed-by: Jason Clinton <jclinton@chromium.org>
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
>
> Change-Id: I02c3d5ea2e65ef852d34a6816d04fe1cad82823a
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775101
> Auto-Submit: Dirk Pranke <dpranke@google.com>
> Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Change-Id: I2c1ae8c60938cbd9316e9075488cc7451068a2f2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2775103
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Auto-Submit: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
This reverts commit 2cf835a9ba.
Reason for revert: doesn't work with depot_tools
error 404 Not found: tools (with applied patch from 2729404)
Bug: 1183975
Original change's description:
> presubmit: Skip owners checks if code-owners plugin is enabled.
>
> If code-owners plugin is enable for the repo, skip owners check on
> commit, and skip checking owners format, as that will be done by
> the plugin.
>
> Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
> Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Change-Id: Id4d560701e4b0144e0aafc5263e09ed6927f6222
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2729409
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Change-Id: I22e6a9d4405d92d26dfbfba64735b6e9ce92674c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2729404
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
If code-owners plugin is enable for the repo, skip owners check on
commit, and skip checking owners format, as that will be done by
the plugin.
Change-Id: I1663baef4f0f27b00423071343fe740f6da50ce7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2727131
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Will be used in a follow-up CL to initialize code-owners client.
Recipe-Nontrivial-Roll: build
Change-Id: Iefe9176320b4d1ae7715e88a8db132e815be76ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2717979
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
This change also adds a target_ref flag to presubmit_support.py.
Recipe-Nontrivial-Roll: build
Change-Id: I6de6bb87fc1482b88d9fbebe5e4ad1dbd8ce9748
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2702792
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
So that presubmit API can use it to check owners
approval.
Change-Id: Ie26c8a83de02b750fd92b4f419fe4d55809e6dbd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2621745
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
logging.warn is deprecated and should be replaced with logging.warning.
Bug:1097402
Change-Id: I55ce007bec47608eeafcf3cd9ac2b90f60073115
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2575621
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
requests.post() creates a new HTTP connection for each call.
It makes presubmit checks slower and timed out.
This CL updates rdb_wrapper to use requests.Session() to use
a connection pool for Sink requests.
Change-Id: Id02ecce85898ae15993d53df17473bb5dfb7e89f
Bug: 1145762
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2532895
Commit-Queue: Scott Lee <ddoman@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Add support for Owners-Override label to allow trusted bots, sheriffs,
and global owners to bypass owners checks.
Bug: 1093627
Change-Id: Ie217369a67e4060807c0a12f84430e30a7c57daa
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2530488
Reviewed-by: Yulan Lin <yulanlin@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Add support for Bot-Review label to allow trusted bots to
bypass owners checks.
Bug: 1093627
Change-Id: Ie7398041e9e32fd87bb415b239b5ec0a4740e1ec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2530516
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Yulan Lin <yulanlin@google.com>
Not outputting a newline when not prompting causes the command prompt to
not appear on a new line.
Change-Id: I791c829e1e0b37c9ad5d6bfdb246db4c779be020
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2380598
Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
This fixes the issue where presubmit_canned_checks issues a warning if
optional arguments in CheckAuthorizedAuthor are not used at all.
This relands commit b09f2bb2f2 with some
additional changes.
R=apolito@google.com
Bug: 1098560
Change-Id: If323d90ab7d6bcca68ed89142ea67edc4be057d2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2373216
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
This reverts commit 149a3b7581.
Reason for revert: warnings are issued if no optional parameters
are passed.
Original change's description:
> Warn when non-inclusive parameters are used
>
> R=apolito@google.com
>
> Bug: 1098560
> Change-Id: Iabc2a5d89a6af1418c1228f3c09eacbbd799b5c7
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2355112
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Josip Sokcevic <sokcevic@google.com>
TBR=apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 1098560
Change-Id: Ib0f188de90121c692f5c0d7271938ee312917fd1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2376042
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Use input_api.version and PRESUBMIT_VERSION to ensure newest versions.
Enable presubmit_support to time/ResultDB wrap each individual
lower-level check in PRESUBMIT.py. The approach taken here is to have
the PRESUBMIT_VERSION = 2.0.0 in PRESUBMIT.py
so that presubmit_support is aware of the new version of PRESUBMIT.py,
then check input_api.version in CheckChangeOnCommit/Upload to ensure
that presubmit_support itself is the new version.
Once these changes are landed, the associated changes in PRESUBMIT.py
(crrev.com/c/2350234) can landed as well.
Change-Id: If6d5e318547bc4f128ffe151387c4dc339ef5dc8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2349979
Commit-Queue: Saagar Sanghavi <saagarsanghavi@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Allows presubmit_support to perform Timing and RDB reporting for each of
the lower-level "CheckXYZ" functions, provided that
ENABLE_NEW_PRESUBMIT_CHECKS is called in the PRESUBMIT.py
file that we are running. I am using the variable to avoid the breakage
caused by the similar previous CL (2332321) which was not backwards
compatible.
In addition, asks for "-realm" with ResultDB so it can maintain the
security boundary.
Bug: 1106943
Change-Id: Ia49e8884e2653bee7de7ed7d254452d55dd5c617
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2341828
Commit-Queue: Saagar Sanghavi <saagarsanghavi@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit dba1d65a19.
Reason for revert: "Check* takes exactly 3 arguments (2 given)" presubmit errors
eg: https://ci.chromium.org/p/chromium/builders/try/chromium_presubmit/933062?
https://crbug.com/1113506
Original change's description:
> [Fixed from older branch] Timing/RDB for lower-level checks
>
> presubmit: Report timing to rdb for individual checks
> Rather than using CheckChangeOnCommit() or CheckChangeOnUpload() as the
> sole entry points for PRESUBMIT.py files, this cl enables presubmit_support.py
> to use any CheckXYZ[OnCommit/Upload] function to serve as an entry point.
>
> This way, we can perform timing and RDB reporting for each of the
> presubmit check functions and have check-specific data.
>
> Bug: 1106943
> Change-Id: Ifdabd68c0904a6d70a828f12de157369c6c1571d
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2332321
> Commit-Queue: Saagar Sanghavi <saagarsanghavi@google.com>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
TBR=dpranke@google.com,estaab@chromium.org,ehmaldonado@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,saagarsanghavi@google.com
Change-Id: Ibd80f70661d2196f903f0c6b79eb74eda6c94d44
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1106943, 1113506
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2340692
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
presubmit: Report timing to rdb for individual checks
Rather than using CheckChangeOnCommit() or CheckChangeOnUpload() as the
sole entry points for PRESUBMIT.py files, this cl enables presubmit_support.py
to use any CheckXYZ[OnCommit/Upload] function to serve as an entry point.
This way, we can perform timing and RDB reporting for each of the
presubmit check functions and have check-specific data.
Bug: 1106943
Change-Id: Ifdabd68c0904a6d70a828f12de157369c6c1571d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2332321
Commit-Queue: Saagar Sanghavi <saagarsanghavi@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Added an rdb wrapper and use it to time CheckChangeOn{Commit, Upload} from every PRESUBMIT.py file, then reports the results to ResultSink.
Bug: http://crbug.com/1107588
Change-Id: I9bc8a18ba615a76f3e0611eae5b2c4adc5924276
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2242097
Commit-Queue: Saagar Sanghavi <saagarsanghavi@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Garrett Beaty <gbeaty@chromium.org>
Some PRESUBMIT.py scripts set DEFAULT_WHITE_LIST and DEFAULT_BLACK_LIST.
This patch adds support to override those defaults during transition
period.
R=apolito@google.com, ehmaldonado@chromium.org
Bug: 1098562
Change-Id: I5dfff253e61485d354a83f3f509d8a6d4d87dec6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2270717
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Slashes and backslashes are interchangeable on Windows in many contexts.
But not all. In particular, if you add a path to a dictionary and then
try looking it up with an equivalent path that has different slashes
then - not surprisingly - the lookup will fail. Therefore it is
important to always use slashes when populating _paths_to_owners. This
is now done, and documented.
Additionally, while fnmatch.fnmatch handles comparing slash separated
paths to backslash separated paths, fnmatch.translate does not. This is
arguably a bug in fnmatch.translate, or at least something that should
be documented. This bug is worked around by having _fnmatch sanitize
file names to always use slashes, and asserting that the patterns do
not contain backslashes.
With these changes this command:
git cl owners --ignore-current --show-all chrome/browser/BUILD.gn
now correctly gives the same results as this command on Windows:
git cl owners --ignore-current --show-all chrome\browser\BUILD.gn
The modified test fails without the other changes, passes with them.
Bug: 1009104
Change-Id: I416c7131281f00e352c1d2b85c30fcc463417fa5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1915410
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 684096347b.
Reason for revert:
Breaks git-cl split.
https://bugs.chromium.org/p/chromium/issues/detail?id=1054888
Original change's description:
> Improve git cl split
>
> This CL changes the behavior of `git cl split` to split the change
> by the size of the resulting CLs. For now, this is based on the number
> of bytes changed, and not by the number of changed lines. Depending
> on the shape of change, this may still produce more CLs than expected
> (and possibly more than before).
>
> A future change will switch the split to be based on the number
> of affected lines, and also introduce a mode to base the split
> on the number of affected files.
>
> Bug: 998922
> Change-Id: I49f868972a61b89b426ef9e2ceedc733eacb4350
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778744
> Commit-Queue: Yannic Bonenberger <yannic.bonenberger@gmail.com>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
TBR=fdoray@chromium.org,dpranke@chromium.org,yannic.bonenberger@gmail.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
Bug: 998922
Change-Id: I466e1245d5f1c934443d850c48c42d3e694e71c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2080205
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Remove input and output stream, that always pointed to sys.stdin
and sys.stdout.
Add tests for DoPostUploadExecuter.
Bug: 1042324
Change-Id: Ib5d092706a242ae97ec877af5dbceef6db7cbb73
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2079411
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
PRESUBMIT.py scripts are executed in presubmit_support.py using exec().
Since PRESUBMIT.py scripts might not be yet compatible with python 3, we
have to execute presubmit_support.py using python 2.
git_cl.py imports presubmit_support.py, and executes presubmit checks using
presubmit_support as a module. This forces git_cl.py to be executed using
python 2 to maintain compatibility for PRESUBMIT.py scripts.
This change allows git_cl.py to be executed using python 3, while
presubmit_support.py is executed using python 2.
Similar changes for post-submit hooks and git-cl try masters will follow.
Bug: 1042324
Change-Id: Ic3bb1c2985459baf6aa04d0cc65017a1c2578153
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2068945
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Also, rename --all-files to --all_files to be consistent with other
options, and remove reviewers, since it's never set in presubmit_support.
Bug: 1042324
Change-Id: I34fa3162c56e39bcdd52ad1e19058357de934f1c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2067351
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
The `author` parameter is never set.
The `local_description` parameter was only set with the
purpose of getting the local description, so separate it into its
own method and call that instead.
Use the absolute root when getting the list of modified files instead.
Bug: 1042324
Change-Id: I8505172edaacee0656dda98ebbd5f8ebfacb3cef
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2038050
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This is a reland of 1c8026d2f1
Original change's description:
> presubmit_support: Add --all-files options.
>
> Will be set when --all is passed to git-cl presubmit to execute
> presubmit checks over all files on a repo.
>
> Bug: 1042324
> Change-Id: I2ebb7c3ec633f9d3a3774314799aa056bc2919c8
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2003071
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Bug: 1042324
Change-Id: I452d98972f9b2c6d0a212029bac5aa6af57dce40
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2038055
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 1c8026d2f1.
Reason for revert:
Description should be set from gerrit before parsing the changes.
Original change's description:
> presubmit_support: Add --all-files options.
>
> Will be set when --all is passed to git-cl presubmit to execute
> presubmit checks over all files on a repo.
>
> Bug: 1042324
> Change-Id: I2ebb7c3ec633f9d3a3774314799aa056bc2919c8
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2003071
> Reviewed-by: Anthony Polito <apolito@google.com>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
TBR=ehmaldonado@chromium.org,apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com
Change-Id: I8c455a269cdd6eb3dea9379faf351a04644bd23f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1042324, 1048885
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2038052
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Will be set when --all is passed to git-cl presubmit to execute
presubmit checks over all files on a repo.
Bug: 1042324
Change-Id: I2ebb7c3ec633f9d3a3774314799aa056bc2919c8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2003071
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Needed by Skia's PostUploadHooks, so that presubmit_support
doesn't depend on git-cl classes.
Bug: 1042324
Change-Id: I02475e346437b26fa8d0d2021ea62754077a2b5b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2007262
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Needed by Skia's PostUploadHooks, so that presubmit_support doesn't
depend on git-cl internal classes.
Bug: 1042324
Change-Id: If8f4e6ea035197667cd9705a4274590c4d11511e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2007670
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This CL changes the behavior of `git cl split` to split the change
by the size of the resulting CLs. For now, this is based on the number
of bytes changed, and not by the number of changed lines. Depending
on the shape of change, this may still produce more CLs than expected
(and possibly more than before).
A future change will switch the split to be based on the number
of affected lines, and also introduce a mode to base the split
on the number of affected files.
Bug: 998922
Change-Id: I49f868972a61b89b426ef9e2ceedc733eacb4350
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1778744
Commit-Queue: Yannic Bonenberger <yannic.bonenberger@gmail.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Only gerrit_url is needed to initialize gerrit_obj.
This will be needed for git cl to call presubmit support
as a script.
Bug: 1042324
Change-Id: I79d78fb67465f989755bf8c8ceb48d665464fa05
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2005050
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
On Windows, scripts on the current directory take precedence over PATH, so
that when testing depot_tools on Windows, calling `vpython.bat` will
execute the copy of vpython of the depot_tools under test instead of the
one in the bot.
As a workaround, we run the tests from the parent directory instead.
Change-Id: I5187943b6cf7499d50b3f1a2ec906a5e8fd84835
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1910448
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
The problems on crbug.com/1017367 should have been fixed by crrev.com/c/1880013
Bug: 1009814
Change-Id: Ia98304ca983b4e7e293fedb9df1fee589de58ba9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1887714
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Exceptions other than OSError are not surfaced.
This caused errors like this to be printed, but not block presubmit,
allowing bugs to sneak in.
Exception in thread Thread-8:
Traceback (most recent call last):
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\presubmit_support.py", line 199, in _WorkerFn
result = self.CallCommand(test)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\presubmit_support.py", line 170, in CallCommand
p = subprocess.Popen(cmd, **test.kwargs)
File "C:\b\s\w\ir\kitchen-checkout\depot_tools\subprocess2.py", line 143, in __init__
super(Popen, self).__init__(args, **kwargs)
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\b\s\w\ir\cipd_bin_packages\cpython\bin\Lib\subprocess.py", line 640, in _execute_child
startupinfo)
TypeError: environment can only contain strings
https://logs.chromium.org/logs/infra/buildbucket/cr-buildbucket.appspot.com/8898840708364523888/+/steps/presubmit/0/stdout
Change-Id: I34e65d8c0050eed7ed26fd782e0a5dc8616f30f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1877051
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
On windows on Python 2, only strings can be environment variables, but
literals are unicode, so we need to convert to strings before setting
env vars.
Change-Id: I40e8e6ae4f5497d56efa568153eb65392dbd85e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1875108
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Some of the modules imported by presubmit_support are unused, and some of those (cPickle, cStringIO) are not available on Python3, so remove them.
Also expose urllib2 via urllib_request and urllib_error to improve compatibility with Python3.
Bug: 1009814
Change-Id: I9f758df4270cc039df092e814c981a7a87b2c350
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1866170
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Ran:
vi $(git grep --name-only iteritems | grep -v third_party)
vi $(git grep --name-only itervalues | grep -v third_party)
vi $(git grep --name-only 'print ' | grep -v third_party)
and edited the files quickly with adhoc macros. Then ran in recipes/:
./recipes.py test train
There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.
Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
Don't overwrite passed kwargs. Make a copy instead.
Otherwise, subsequent calls with the same kwargs will have
stdin set to subprocess.PIPE.
Bug: 984182
Change-Id: I358ffa1951e8b42486e0ac3a0d3d587a93c6dc4e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1769405
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Add run_on_python2 and run_on_python3 to allow to be tests on
either only python2, only python3 or both.
Bug: 984182
Change-Id: I87d3b67412129cf3a8e627cd6bc97daa81147657
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1758849
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: smut <smut@google.com>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Josip Sokcevic
Bruce Dawson
Louis Romero
Andrew Grieve
Aravind Vasudevan
Sean McAllister
Tom McKee
kyle Ju
Dirk Pranke
Greg Thompson
Allen Webb
Erik Staab
Gavin Mak
Edward Lesmes
Edward Lesmes
Stephen Martinis
Scott Lee
Garrett Beaty
Saagar Sanghavi
Ben Pastene
Aaron Gable
Yannic Bonenberger
Marc-Antoine Ruel
Dan Beam