When hooks multiple cipd packages under same directory, gclient would
pop up error that same directory name appears multiple times in deps
section. This CL overrides verify_validity for CipdDependency to
support multiple packages hooked in same directory.
Bug:812386
Change-Id: Ia4f1fe0e3c8481c9b06c1d22b6e98d98e1e4c309
Reviewed-on: https://chromium-review.googlesource.com/920686
Commit-Queue: Shenghua Zhang <shenghuazhang@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Previously, it counted the number of UTF-8 bytes when spacing out the
table, not the number of code points.
Bug: 808905
Change-Id: Ice5504089e0f7097e108c6dfbbb810620b9dfc94
Reviewed-on: https://chromium-review.googlesource.com/901142
Commit-Queue: Matt Giuca <mgiuca@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Although we want to prevent dfgs from untar'ing files to a parent
or sibling of its target directory, normal files that just happen
to have ".." in their name (i.e. not preceding a path separator) are
okay.
R=hinoka
Bug: 807286
Change-Id: Ibdc2c3615c4778ef66abceb532a4f671fbdab8ef
Reviewed-on: https://chromium-review.googlesource.com/912430
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Previously, if you used custom metadata keys in a test commit (like
GitRepo.AUTHOR_NAME), they would also be treated as filenames, and fail
because they aren't strings. (This wasn't a problem because it isn't
currently used in any tests.)
Bug: 808941
Change-Id: Id7c4fa5822741925beba591ea587bd8ebbf2e478
Reviewed-on: https://chromium-review.googlesource.com/901122
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
The purpose of this CL is to remove confusion around whether or
not git-cl support the similarity flags when talking to Gerrit.
It does not. However, because Rietveld support has not been
fully ripped out of git-cl, this change does modify some of the
Rietveld-specific code to remove similarity support from everywhere.
Bug: 800902
Change-Id: I8703ed60d6889c7a36400b8a9f8f26de4669020a
Reviewed-on: https://chromium-review.googlesource.com/912389
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Also, some lines went over 80chars, so I ran 'yapf --style chromium' which
formatted some unrelated things.
BUG=905285
R=dpranke
Change-Id: Iee5f46d88a6e9782612cc4f9e5a2cb72d62ab6af
Reviewed-on: https://chromium-review.googlesource.com/907736
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
The first call to gsutil causes a network call and makes the download
script slow also in the most optimal cases. This CL refactors the
download script and moves the first gsutil call after checking locally
if sha1s match.
1) This turns the input acquisition into a generator and buffers the
files and sha1s in a list before multithreading.
2) This sequentially checks the sha1s and files and bails out early if
all match. In Chrome-land, we usually call this script with only one
file. There are some cases with around 4. This could also be
parallelized if the need arises.
3) The initial gsutil check, which ensures gsutil is updated, is moved
right in front of the multithreaded downloads.
The performance of one call to download_from_google_storage for an
existing 500MB file is 2.3s before this CL and 1.2s after (most of the
remaining time left is spent for making sha1sum).
Example for full gclient runhooks (when everything is up-to-date):
Chromium: 32s before, 12s after
V8: 12s before, 3s after
Bug: 776311
Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
Reviewed-on: https://chromium-review.googlesource.com/897562
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Flattening currently ignores custom_vars, but it looks like that was
just an attempt to avoid including all the runtime-calculated values
(e.g. the host OS) in the flattened output, and not because there's any
real issue with supporting explicit overrides.
This is useful for changing default values in the flattened buildspecs
(e.g. to enable internal checkouts by default).
R=dpranke@google.com, iannucci@google.com
Bug: 570091,807318
Change-Id: I6d743f33cf7e3e202d5e68d21657f74f0e4c001b
Reviewed-on: https://chromium-review.googlesource.com/895927
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
This will allow us to show the progress when cloning a new repo when running
commands like gclient sync.
R=agable@chromium.org
Bug: 722686
Change-Id: I268cba343ea4c3c024292c9341d5009aa112b184
Reviewed-on: https://chromium-review.googlesource.com/890524
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
I don't know of any use-case where someone would want to
use git-cl-patch to pull a (e.g.) chromium change into
their (e.g.) v8 checkout.
Bug: 803918
Change-Id: Id53f1cc3ab97e623d0098bb366a573b18b54e91a
Reviewed-on: https://chromium-review.googlesource.com/876930
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Suppose the final paragraph of a commit message looks like this:
"""
And here's the final paragraph.
Bug: 1234
TBR=soandso
Change-Id: deadbeef
"""
In this case, we don't want to lose the Bug and Change-Id footers,
so we process the whole final paragraph. *But* we'd also like to
help the user get things formatted correctly. This change lets
git_footers notice this situation, and insert a newline before the
first well-formed footer (Bug: in this case), so that the set of
well- and mal-formed footers are separated from the rest of the
malformed body text.
In the rare case where the last line of the last non-trailer paragraph
is a url, this will also visibly push the url into the block of
trailers (where it doesn't belong), prompting the user to fix it.
A more comprehensive fix for that particular case is coming later.
Bug: 766234
Change-Id: I6ae0072fff68ddf06e6f43b70f9a82a7f247f4ab
Reviewed-on: https://chromium-review.googlesource.com/849481
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
It's unlikely that anyone would ever intend to have a git trailer
whose key is "http", but since URLs are often long, it is rather
likely that someone would put a URL on a line all by itself. When
that happens, don't accidentally interpret it as a footer.
R=iannucci, tandrii
Bug: 766234
Change-Id: I3101119c4e49e20339487618cc1719452b026d90
Reviewed-on: https://chromium-review.googlesource.com/849484
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This includes a minor refactor so that some gclient_scm methods
can all share the same core.quotePath specifier.
R=iannucci
Bug: 792302
Change-Id: Iaadf190f5c0666787cf7c2ccda88d6dba9aace9b
Reviewed-on: https://chromium-review.googlesource.com/823131
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
This is a reland of 9219d35688
The original was reverted due to a typo (core,quotePath instead
of core.quotePath). This version is fixed.
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
>
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I28d2260948aaf63bd865888c2f60e4cdee9aea48
Reviewed-on: https://chromium-review.googlesource.com/822990
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Context: The WPT importer consumes the JSON produced by
git cl try-results is consumed and uses it to determine
when the CQ has passed (whether all non-experimental cq
jobs were successful) and where layout test results are
stored.
Added fields:
created_ts: This could be used to sort builds by time.
tags: Includes user_agent tag and experimental tag, as well as
experimental: whether the job is experimental
Bug: 792652
Change-Id: Ic6099a21b21db668b88b1a8e54aefd84529a2a91
Reviewed-on: https://chromium-review.googlesource.com/819573
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
This reverts commit 9219d35688.
Reason for revert: unfortunately this says "core,quotePath" and since it includes recipe changes, we need something that the roller can munch on :(
Original change's description:
> Use core.quotePath=false when git is listing files
>
> This prevents git from putting quotes around some file names
> (those that have astral-plane characters) and not around others.
>
> R=maruel
>
> Bug: 792302
> Recipe-Nontrivial-Roll: build
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
> Reviewed-on: https://chromium-review.googlesource.com/815454
> Commit-Queue: Aaron Gable <agable@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
TBR=maruel@chromium.org,agable@chromium.org
Change-Id: I226388f19024403240a1443eb2b878b9293220e1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 792302
Reviewed-on: https://chromium-review.googlesource.com/821671
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
This prevents git from putting quotes around some file names
(those that have astral-plane characters) and not around others.
R=maruel
Bug: 792302
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I3b6a6b36c4720116de811b40177b59aa25c263db
Reviewed-on: https://chromium-review.googlesource.com/815454
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
If a commit subject contains [hashtags], include them in Gerrit CL.
Replace consecutive non-alphanums with dash.
Also add --hashtag flag for explicit hashtagging.
Bug:
Change-Id: I25aed286013043263f959ff340a5b5478faa0f27
Reviewed-on: https://chromium-review.googlesource.com/764974
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Sometimes, InputApi users need to create temporary files, write to them and
pass them to another process, like this:
with input_api.tempfile.NamedTemporaryFile() as f:
f.write('foo')
input_api.subprocess.check_output(['/path/to/script',
'--reading-from', f.name])
While this works fine on Unix, on Windows subprocess cannot open and read
the file while we have it open for writing.
To work around this, we now offer a CreateTemporaryFile() that wraps a call
to tempfile.NamedTemporaryFile(delete=False), and we then take care of
removing all files created this way at the end of a presubmit run.
The idea is for users to do something like this:
with input_api.CreateTemporaryFile() as f:
f.write('foo')
f.close()
input_api.subprocess.check_output(['/path/to/script',
'--reading-from', f.name])
with the temporary file being removed automatically in a transparent fashion
later.
Bug: 780629
Change-Id: I0d705a5d52928a43f39a51f94a2c48d277bd5ced
Reviewed-on: https://chromium-review.googlesource.com/758637
Commit-Queue: Raphael Kubo da Costa (rakuco) <raphael.kubo.da.costa@intel.com>
Reviewed-by: Aaron Gable <agable@chromium.org>
A CL might delete an OWNERS file which we still need to read for the OWNERS
check.
BUG=778870
R=agable@chromium.org
Change-Id: I25636ff36228a1afb3c10edf5c2419773a4d057e
Reviewed-on: https://chromium-review.googlesource.com/754623
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Jochen Eisinger <jochen@chromium.org>
In Python land, the method run once a test method runs is called tearDown(),
not cleanUp(). In other words, the existing methods were never called and we
were always leaving lots of "gstools_test*" directories in /tmp.
Change-Id: Ib1de95c7ba92922b98571780f389237de0dcb253
Reviewed-on: https://chromium-review.googlesource.com/753383
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This check will be used in Chromium, V8 and Catapult PRESUBMIT scripts.
R=maruel@chromium.org, tandrii@chromium.org
Bug: 777893
Change-Id: I2ca1e774b89787c4d3b5f336315d145571858864
Reviewed-on: https://chromium-review.googlesource.com/738169
Commit-Queue: Sergiy Byelozyorov <sergiyb@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
This reverts commit 47b67c426b.
Reason for revert: Bug in bot_update.py resource, breaks recipe roller.
Original change's description:
> Reland "bot_update recipe: Upload source manifest"
>
> This reverts commit c3d1208d5c.
>
> Also:
> * Instead of replacing "manifest", just add a new "source_manifest"
> to the output JSON. This allow transition without breakage.
> * Change the test api so test for recipe output changes.
>
> The plan is to land this first, switch all downstream to "source_manifest",
> and then remove the original "manifest" key.
>
> Bug: 772529,776299
> Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: release_scripts
> Recipe-Nontrivial-Roll: skia
> Recipe-Nontrivial-Roll: skiabuildbot
> Recipe-Nontrivial-Roll: build
> Reviewed-on: https://chromium-review.googlesource.com/731378
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Ryan Tseng <hinoka@chromium.org>
TBR=iannucci@chromium.org,hinoka@chromium.org
Change-Id: I7a4ee904075e8b75b8a47f9ef0cd8a633af85a9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 772529, 776299
Reviewed-on: https://chromium-review.googlesource.com/748312
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
This reverts commit c3d1208d5c.
Also:
* Instead of replacing "manifest", just add a new "source_manifest"
to the output JSON. This allow transition without breakage.
* Change the test api so test for recipe output changes.
The plan is to land this first, switch all downstream to "source_manifest",
and then remove the original "manifest" key.
Bug: 772529,776299
Change-Id: Iffb75f18046f8e4c058afe077872d4257b9dd754
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: release_scripts
Recipe-Nontrivial-Roll: skia
Recipe-Nontrivial-Roll: skiabuildbot
Recipe-Nontrivial-Roll: build
Reviewed-on: https://chromium-review.googlesource.com/731378
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
the tests haven't been ran by presumbit for a while because of the plural in
the filename.
At some point some post "gsutil cp" file checking happened, which
broke the tests. This adds a callback to the fake gsutil cp so
that the expect file is copied over.
This also removes "gsutil version" checking from gsutil.py
and just assume that if the file exists, then it's good, which
should shave about 1-2s off of each gsutil.py call.
Bug: 772740,776311
Change-Id: I4fef62cfd46a849afed1f095dd6a96069376d13d
Reviewed-on: https://chromium-review.googlesource.com/707758
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Shenghua Zhang
Nodir Turakulov
John Budorick
Matt Giuca
Andrii Shyshkalov
Aaron Gable
Tom Anderson
Edward Lemur
Michael Moss
Michael Achenbach
Lei Zhang
Quinten Yearsley
Robbie Iannucci
Mun Yong Jang
Daniel Cheng
Raphael Kubo da Costa
Nick Carter
Jochen Eisinger
Sergiy Byelozyorov
Ryan Tseng
Ryan Tseng
Dirk Pranke