Include a luci-auth check of the account logged in.
This is to check that a user isn't using a non-google account on a corp machine. Also this check is only performed every 12 hours, so it doesn't impact every build. This check was already performed with other auth mechanisms, this is just making sure it is being done for luci-auth as well.
Bug: b/330339907
Change-Id: I8ea97c8de0f2d74c2d735fa959d8227cec35df46
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379037
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@google.com>
This CL adds a "early terminate the field based on field value" parser
mechanism to end the field as soon as the field value provides an
unambiguous answer to the question we care about.
This is to prevent over-extraction over certain fields (specifically,
local modifications) which can either be a definitive answer (e.g. No
modification) or multi-line free-form texts (which may contain unknown
fields that we don't care about at this stage).
This mitigates over extraction of README.chromium files like:
```
Local Modifications:
None
How to Uprev:
Steps...
```
Where the old parser would extract "None\n\nHow to Uprev:\nSteps..."
This CL also refactors single line fields to use the same early
termination mechanism since single line field simply ends as soon as
the line is parsed.
Union[Something, None] is changed to Optional[Something] based on
styleguide.
Bug: b/324149233
Change-Id: I3fca80eaceb071263f8ae8730afda230fff0bbb0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5394917
Reviewed-by: Anne Redulla <aredulla@google.com>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
This script allows users to upload one file or one or more compressed
directories to GS. It will also output the DEPS blob with the required
sha256sum and size_bytes.
Verified script locally.
Bug: b/328065301
Change-Id: Ic8e894917eb4efde4f287ffb6869a22be1e89bb3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381863
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This utilizes the same tarfile validation as
download_from_google_storage, but checks that the links and contents
are inside the tarfile's multiple top level directories instead of
just one directory inside the tarfile.
Bug: b/324418194
Change-Id: I73505c72084d54ca8088213dbc40b371675f8dc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381117
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Paths containing ':' characters are illegal only on Windows platforms.
So, ssh_url like git@github.com:chromium/chromium.git will generate
runtime exception on Windows. We replace ':' with '__' to avoid
exceptions in the ssh form of git_url case.
R=ddoman@chromium.org
Bug: 329781388
Change-Id: Ia4db6a0927b40b521477e5d90addb4056c2c8ef8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5369296
Auto-Submit: 吴金立 <wujinli.cn@gmail.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
_process_diff assumes "@@" will always be present in a diff, but those
aren't present in a git diff where only the file mode changed.
Bug: b/323243527
Change-Id: Id129bc6c14994affc43a77feab553fa119612b7b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384645
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This CL makes scan.py to sort result summary (keyed on the type of
validation results), so the output summary will have a predictable
order for easier diffing.
Change-Id: I0eda8a439e159eae667729512424b813eb0f5e11
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5390925
Auto-Submit: Jiewei Qian <qjw@chromium.org>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Anne Redulla <aredulla@google.com>
Since macOS 11, the underlying python `platform.mac_vers()` call
that mac_release is populated from returns "10.16" on all versions
of macOS.
Switch this to just shell out to `sw_vers` to pick up the current
platform version instead.
Non-trivial downstream rolls are because `api.platform.mac_release`
no longer mocks the version of the SDK installed... however it would
be better to set this by just using the explicit sdk_version property.
R=yueshe@google.com
Bug: 331252586
Recipe-Nontrivial-Roll: build_limited
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Change-Id: I41a27383f67db4835ba84b7a47813884879818cf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5394351
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Yue She <yueshe@google.com>
There may be other tests which aren't correct though, this is just
the one I saw during presubmit.
R=aravindvasudev@google.com
Bug: 330347045
Change-Id: I6499b5103bf15c30f5e8a2fdcd7abeba094a39d7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5394357
Auto-Submit: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
This makes it much faster to get a diff for a set of multiple files.
Bug: b/323243527
Change-Id: I6c54a4fa8782d4a0bbb187b8bed4fbd52be47344
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384647
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
This CL adds a "structured" concept to the parser. In a structured
field, the parser will proactively look for field-like patterns to
start a new field (even if they aren't known fields).
This mitigates the issue when an unknown field immediately
follows a multi-line text field, such as:
URL: https://example.com
UnknownField: abc
And URL field value parses to
"https://example.com<newline>UnknownField:abc".
Bug: b/324149233
Change-Id: I54807bd7b242fc14c679483453ade83f8fd20225
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379679
Reviewed-by: Anne Redulla <aredulla@google.com>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
This is a reland of commit 4d2728b03e
Added default values for host and project in case input.gerrit does not
exist.
Original change's description:
> Include host and project in code-owners not enabled error.
>
> Bug: 329231688
> Change-Id: I36bb24f167854f40718ec49559d9aaa640fa1a44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384365
> Auto-Submit: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Joanna Wang <jojwang@chromium.org>
> Reviewed-by: Scott Lee <ddoman@chromium.org>
> Commit-Queue: Scott Lee <ddoman@chromium.org>
Bug: 329231688
Change-Id: If65e362d0aba97d804a75c078452bab8152ae224
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384366
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
This reverts commit 4d2728b03e.
Reason for revert: Won't work if gerrit=None
Original change's description:
> Include host and project in code-owners not enabled error.
>
> Bug: 329231688
> Change-Id: I36bb24f167854f40718ec49559d9aaa640fa1a44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384365
> Auto-Submit: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Joanna Wang <jojwang@chromium.org>
> Reviewed-by: Scott Lee <ddoman@chromium.org>
> Commit-Queue: Scott Lee <ddoman@chromium.org>
Bug: 329231688
Change-Id: I49d23611b62f1979578ae52861a2007a1d34778e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5381788
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Explicitly sorts the discovered metadata file, and defines
ordering of validation messages, so the tools will output
will be self-consistent.
Change-Id: I9b263a16b151c014e5950638f066376469c701df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379678
Commit-Queue: Jiewei Qian <qjw@chromium.org>
Reviewed-by: Anne Redulla <aredulla@google.com>
Bug: 329231688
Change-Id: I36bb24f167854f40718ec49559d9aaa640fa1a44
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5384365
Auto-Submit: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
Commit-Queue: Scott Lee <ddoman@chromium.org>
Occasionally on windows there seems to be a racing problem where rmtree
tries to delete something that no longer exists. This seems to happen
when a build is cancelled/terminated while in-progress, so racing
actions in reproxy might have deleted the temporary file while autoninja
is cleaning up stuff. In any case, missing a file in this case isn't
something to fail over.
Bug: b/329738642
Change-Id: I86240e31093c6415707878277b01db5931b175ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376532
Reviewed-by: Ben Segall <bentekkie@google.com>
Commit-Queue: Michael Savigny <msavigny@google.com>
os.walk is more efficient than the current handwritten traversal.
Measured the time to scan reduced from 30s+ to 8s on p920 on
chromium/src.
`followlinks=True` is set to preserve behavior that os.path.isdir
returns True for symlink to directories, and the current traversal code
will descend into those.
Change-Id: I941eec9105a46d6538ca484fbb5249a75888e38a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379945
Reviewed-by: Anne Redulla <aredulla@google.com>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
If is_first_class_gcs file exists with contents '1'
- GCS hooks will download content and then overwrite the file with '0'
- First class GCS deps will download as necessary by checking the hash
file.
If there is no is_first_class_gcs file or its content is '0' for a GCS hook:
- Hook will run normally and, if content is downloaded, an is_first_class_gcs
file will be written with '0'
If there is no is_first_class_gcs file or its content is '0' for a first
class gcs dep:
- Content will be downloaded and is_first_class_gcs will be set to '1'
Verified locally with migrating and unmigrating a GCS hook.
TODO: update other GCS hook scripts in chromium/src to check migration
file.
Bug: b/324418194
Change-Id: Ida8541cb70839b86e99115bcabfc38bc87e40136
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5368250
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
This flag allows a diff to be generated based on an upstream commit
for provided files and for checks to be run with on that diff.
Bug: b/323243527
Change-Id: Ia3f4ec50b15eabb90d391b9bc795fc4b35a35a08
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5376402
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Create a script to create a unified git diff of a local workspace
against a remote Gerrit commit. This will be used to create a diff file
for running presubmits.
Bug: b/323243527
Change-Id: Ia0d287624162dbfe5231a5653c9a962a6c85c8e1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5374833
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
GCS deps will require a size_bytes field of type `int`. If gclient
sees that the actual byte size is different than the noted size, it
will raise an exception.
Bug: b/328065441
Change-Id: If61496eae39cd372966d7e64ba4a84759708c60b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5367540
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
I initially didn't want to make this code Google-specific, however I now
believe that it's better for everyone to simplify it like this:
- It's much less code to just check for the path prefix. We do this
already in gclient.py, too, so the approach is proven to work.
- The call to findmnt introduces a 15ms overhead that affects everyone,
even users who will never have their workspace on Cog.
- External folks who run their own infrastructure likely have their own
fork of depot_tools anyway and can patch this as needed.
Bug: b/324547324
Change-Id: Ic00d417c280ac71529d106cfb2bcc844dfd44bd3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5358773
Commit-Queue: Philipp Wollermann <philwo@chromium.org>
Auto-Submit: Philipp Wollermann <philwo@chromium.org>
Reviewed-by: Junji Watanabe <jwata@google.com>
This doesn't catch all the variants of --help that getopt allows but it
does catch --help, which is probably the most common.
Bug: 329317224
Change-Id: I2dde63df84bd2c2ade5f9d2c1ab10c3dc13fba70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370829
Auto-Submit: David Baron <dbaron@chromium.org>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
google-java-format will only change the lines given by --lines, which
means that:
a) Unused imports are not removed (assuming they were existing imports)
b) Excessive blank lines between methods are not removed
To address this, run google-java-format a second time with
--fix-imports-only, include deleted lines in ranges, and expand ranges
by 2 lines on each side.
Bug: 1462204
Change-Id: Ia12af6def28d9fe11e8315a951d45908f7cee725
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5362198
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Notes that --offline is for disabling remote execution in general not just for goma.
Bug: b/329317224
Change-Id: I5538c9fac6d5161e98e1d156e61c512c6a1e8496
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5370094
Auto-Submit: Michael Savigny <msavigny@google.com>
Reviewed-by: Fumitoshi Ukai <ukai@google.com>
Commit-Queue: Fumitoshi Ukai <ukai@google.com>
When an 'output_file' arg is included, the downloaded object from GCS
will be downloaded in that path.
Example:
```
'src/buildtools/linux': {
'bucket': 'bucket123',
'object_name': 'clang-format-version123',
'dep_type': 'gcs',
'sha256sum': 'abcd123',
'output_file': 'clang-format',
}
```
The GCS object will be downloaded at src/buildtools/linux/clang-format
Bug: b/324418194
Change-Id: I1049abeb09a1027c5477d955e50611d43015d0a4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5353387
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
The check is intended to ensure paths don't traverse outside
the extracted directory. However, the check was too strict: it
banned all links, even relative links that still point inside the
target directory. This CL relaxes the requirement to allow valid
links. This is required to allow library symlinks for the
instrumented libraries. Examples:
libpcre.so -> libpcre.so.3.13.3
libpcre.so.3 -> libpcre.so.3.13.3
libpixman-1.so -> libpixman-1.so.0.38.4
libpixman-1.so.0 -> libpixman-1.so.0.38.4
libpng16.so -> libpng16.so.16.37.0
libpng16.so.16 -> libpng16.so.16.37.0
Bug: 320564950
Change-Id: I2aae18b86b1f1cc3d73a1b80c06d757af782f700
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5369132
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Auto-Submit: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Joanna Wang <jojwang@chromium.org>
A recent change added --diff support for Java files in git cl format,
but unfortunately it broke the behavior when both --dry-run and --diff
are passed, and made git cl format indicate that all changed files need
to be reformatted.
Fix so that --diff overrides --dry-run when running google-java-format,
as passing --dry-run makes the formatter only output file names which
then confused the diff creation code. This makes behavior for Java in
line with C++ where --dry-run doesn't change much if --diff is used.
Change-Id: I7f8eecb13064a0046935489e6c5cf3af566f7f56
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5357818
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Tomasz Śniatowski <tomasz.sniatowski@xperi.com>
recipe-roller
Joanna Wang
Michael Savigny
Jiewei Qian
Stephanie Kim
Jinli Wu
Takuto Ikuta
Gavin Mak
nqmtuan
Robert Iannucci
Brian Ryner
Josip Sokcevic
Joanna Wang
Josh Berenhaus
Philipp Wollermann
L. David Baron
Andrew Grieve
Tom Anderson
Tomasz Śniatowski