file_list can be none if gclient runs with --nohooks.
Fix apply_patch_ref so that it doesn't try to update file_list when it is
None.
TBR=agable@chromium.org
Bug: 643346
Change-Id: If00547da004415edfe68196a44cbda753b4db017
Reviewed-on: https://chromium-review.googlesource.com/989279
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Add support for more ways to express revisions in DEPS files, notably
"<origin-url>@<revision>" which is needed for src-internal.
Bug: 760633
Change-Id: I86724b9c077c6101a0c424d80460123d15533da8
Reviewed-on: https://chromium-review.googlesource.com/988493
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
The URLs for the dependencies in DEPS files might end in '.git', so
search for them too.
R=agable@chromium.org
Bug: 643346
Change-Id: I955c2750179b54fec26423f5123b3fa7eea38d96
Reviewed-on: https://chromium-review.googlesource.com/987630
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This is a reland of 88f9c40e0c
We no longer need the vars dict to be declared before vars can be used.
This was causing problems because gclient flatten printed vars after deps,
breaking the above assumption.
Original change's description:
> gclient eval: Expand vars while parsing DEPS files
>
> Introduce a Parse function that takes care of expanding vars while parsing
> the DEPS file.
>
> It wraps Exec and exec calls, and supports deferring the expansion until
> later, so gclient flatten gets access to the unexpanded version.
>
> Bug: 821199
> Change-Id: I943b021cc4474c9cda67b3816b841dd8ada3f5b2
> Reviewed-on: https://chromium-review.googlesource.com/973749
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Bug: 821199
Change-Id: I583df23558f91871e1a2aa2574c20d35e54635f6
Reviewed-on: https://chromium-review.googlesource.com/981086
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This will make it easier to programmatically determine where a flattened
entry originated, which is needed for recursively branching and
unpinning DEPS files.
BUG=825063
R=dpranke@google.com
Change-Id: Id280c0b0a95b8664602e0ec4513722fe4d6d1ebf
Reviewed-on: https://chromium-review.googlesource.com/977326
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
This adds a --enable-gclient-experiment flags that tells bot_update.py
to skip applying the patch, and instead forward the flags to gclient.
Bug: 643346
Change-Id: Ia4275a126e6adba54dfcc894d224c50c166db90e
Reviewed-on: https://chromium-review.googlesource.com/962938
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This is a reland of 88f9c40e0c
Original change's description:
> gclient eval: Expand vars while parsing DEPS files
>
> Introduce a Parse function that takes care of expanding vars while parsing
> the DEPS file.
>
> It wraps Exec and exec calls, and supports deferring the expansion until
> later, so gclient flatten gets access to the unexpanded version.
>
> Bug: 821199
> Change-Id: I943b021cc4474c9cda67b3816b841dd8ada3f5b2
> Reviewed-on: https://chromium-review.googlesource.com/973749
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Bug: 821199
Change-Id: Ic04af0cae59abc01a0382e2de3497a91cf7e62fd
Reviewed-on: https://chromium-review.googlesource.com/978561
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 88f9c40e0c.
Reason for revert: suspected to be causing update scripts failures and prevent gce bots from connecting
https://build.chromium.org/deprecated/tryserver.chromium.win/builders/win7_chromium_rel_ng/builds/128479/steps/update_scripts/logs/stdio
Original change's description:
> gclient eval: Expand vars while parsing DEPS files
>
> Introduce a Parse function that takes care of expanding vars while parsing
> the DEPS file.
>
> It wraps Exec and exec calls, and supports deferring the expansion until
> later, so gclient flatten gets access to the unexpanded version.
>
> Bug: 821199
> Change-Id: I943b021cc4474c9cda67b3816b841dd8ada3f5b2
> Reviewed-on: https://chromium-review.googlesource.com/973749
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
TBR=agable@chromium.org,dpranke@chromium.org,ehmaldonado@chromium.org
Change-Id: Ib9b84c13ef9b56735fd8f714b74a6601c61715e5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 821199
Reviewed-on: https://chromium-review.googlesource.com/976721
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Introduce a Parse function that takes care of expanding vars while parsing
the DEPS file.
It wraps Exec and exec calls, and supports deferring the expansion until
later, so gclient flatten gets access to the unexpanded version.
Bug: 821199
Change-Id: I943b021cc4474c9cda67b3816b841dd8ada3f5b2
Reviewed-on: https://chromium-review.googlesource.com/973749
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Currently in LUCI `git cl` is invoked on chromium_presubmit using vpython. This
means that it operates under the depot_tools .vpython environment (which is
blank).
Some features in presubmit_support allow invocation of python subprocesses, and
previously they would use `sys.executable` (especially on windows, but
occasionally on non-windows as well).
In non-vpython environments, this just picks up the system python and whatever
modules happen to be installed there. Some python unittests work around this on
non-windows systems by having a shebang line which explicitly invokes vpython.
This CL changes presubmit_support so that invocations of 'python', or executions
of '.py' scripts will always end up invoking vpython. In the best case, this will
pick up the invoked scripts' vpython environment. In the worst case, this will
invoke the script under an empty vpython environment (aka "stock python").
R=dpranke@chromium.org, jbudorick@chromium.org, tandrii@chromium.org, tikuta@chromium.org
Bug: 821669
Change-Id: I5d2d5dfd0364022d56833c2c8af4983553a29c7a
Reviewed-on: https://chromium-review.googlesource.com/961865
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Mimics bot_update's functionality to apply gerrit refs in gclient
via --gerrit-ref flags.
When the patch fails to apply, gclient sync will return exit code 2.
The idea is to move this logic from bot_update to gclient sync to
deal when patches for projects like ANGLE are tried on Chromium bots.
This way the patch is applied before recursively parsing and syncing
ANGLE’s DEPS.chromium file, which doesn't currently happen.
Bug: 643346
Change-Id: I7e2018b3c393a5ac9852b8c3611f906977eeeb18
Reviewed-on: https://chromium-review.googlesource.com/961605
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This reverts commit 33bf49538c.
Reason for revert: Done debugging.
Original change's description:
> Add debugging info when gclient deletes an old DEPS entry.
>
> To help figure out why it decided to do so.
>
> BUG=823586
>
> Change-Id: I93d7c9f7af6145ee0ebd9f5ad4483f27925e84d6
> Reviewed-on: https://chromium-review.googlesource.com/970082
> Commit-Queue: Lei Zhang <thestig@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
TBR=thestig@chromium.org,agable@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 823586
Change-Id: I476cbe423b849582f8684426653f0c08062e89e9
Reviewed-on: https://chromium-review.googlesource.com/974458
Reviewed-by: Lei Zhang <thestig@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>
Since we only support Var, and Var only does one thing, we hard code that
into the parser. Then, we don't need global_scope.
local_scope hasn't been needed for a while.
Bug: 760633
Change-Id: I21b171a8b71e7dcaf8e29c780ca88b9f46f368e8
Reviewed-on: https://chromium-review.googlesource.com/972611
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
This is a reland of 7f4c905fc5
Original change's description:
> gclient: Add commands to edit dependencies and variables in DEPS
>
> Adds 'gclient setvar' and 'gclient setdep' commands to edit variables
> and dependencies in a DEPS file.
>
> Bug: 760633
> Change-Id: I6c0712cc079dbbbaee6541b7eda71f4b4813b77b
> Reviewed-on: https://chromium-review.googlesource.com/950405
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
Bug: 760633
Change-Id: Ia46c74d02e5cc3b67517dfa248f597cb3d98ef3d
Reviewed-on: https://chromium-review.googlesource.com/969457
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
gclient previously used a CIPD root per DEPS file. This didn't work
in cases in which a DEPS file wanted to specify a CIPD package outside
of its directory hierarchy, though, as is the case with buildspecs.
Bug: 755920
Change-Id: I0d6c3db567f17f9171c0feaaf9ed6bc64db22757
Reviewed-on: https://chromium-review.googlesource.com/955933
Commit-Queue: John Budorick <jbudorick@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
To help figure out why it decided to do so.
BUG=823586
Change-Id: I93d7c9f7af6145ee0ebd9f5ad4483f27925e84d6
Reviewed-on: https://chromium-review.googlesource.com/970082
Commit-Queue: Lei Zhang <thestig@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This helps make it slightly easier to understand.
Change-Id: Iaf68409374be67e03d5708b409c1ad9229c68b1d
Reviewed-on: https://chromium-review.googlesource.com/970161
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>
This reverts commit 7f4c905fc5.
Reason for revert:
When running "gclient sync" on a V8 checkout, it says:
File "/work/chrome/depot_tools/gclient.py", line 995, in run
self.ParseDepsFile()
File "/work/chrome/depot_tools/gclient.py", line 874, in ParseDepsFile
self._postprocess_deps(deps, rel_prefix), use_relative_paths)
File "/work/chrome/depot_tools/gclient.py", line 660, in _postprocess_deps
dval['condition'], self.condition)
TypeError: '_NodeDict' object does not support item assignment
on the recursive descent into third_party/android_tools/DEPS. Any chance that's related to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/950405 ?
Original change's description:
> gclient: Add commands to edit dependencies and variables in DEPS
>
> Adds 'gclient setvar' and 'gclient setdep' commands to edit variables
> and dependencies in a DEPS file.
>
> Bug: 760633
> Change-Id: I6c0712cc079dbbbaee6541b7eda71f4b4813b77b
> Reviewed-on: https://chromium-review.googlesource.com/950405
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
TBR=agable@chromium.org,ehmaldonado@chromium.org
Change-Id: If58f6b15d31b19fc53294f1e41d26b4e684a2cf9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 760633
Reviewed-on: https://chromium-review.googlesource.com/969165
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Adds 'gclient setvar' and 'gclient setdep' commands to edit variables
and dependencies in a DEPS file.
Bug: 760633
Change-Id: I6c0712cc079dbbbaee6541b7eda71f4b4813b77b
Reviewed-on: https://chromium-review.googlesource.com/950405
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This method hasn't been supported for a long time now (and besides, its
only for Rietveld).
R=agable@chromium.org, tandrii@chromium.org
Recipe-Manual-Change: infra
Change-Id: Ie6e63834dca67962db29f2cb407950ed85db55a7
Reviewed-on: https://chromium-review.googlesource.com/957832
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
e.g. when specifying both '--revision src@<something>' and
'--revision <src_gerrit_repo>@<some_gerrit_ref>', gclient will sync to
'<some_gerrit_ref>' instead of '<something>'.
This is useful when specifying which gerrit change to patch, which
should take priority over other revision specifications.
Bug: chromium:643346
Change-Id: Ibc21ede355b56e4da966f38f144ce6f6f1743403
Reviewed-on: https://chromium-review.googlesource.com/949981
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Add a --output-json flag to gclient revinfo so is easier for other
programs to process its output.
Bug: None
Change-Id: Ic120e7a279f3ed537ead624ab9bfd1f141485487
Reviewed-on: https://chromium-review.googlesource.com/952206
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
This allows people to configure their gitcookies file to be
whereever they want. As long as it actually exists and has
credentials in it, we'll accept it, just like git itself.
Change-Id: I4aa4806ddca0e61b28b003b0d3bc486407c13ab4
Reviewed-on: https://chromium-review.googlesource.com/951917
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Add --path and --url options to show only the dependencies that match
one of the given paths or URLs.
Bug: None
Change-Id: I12c205545b7ce54b56abcd62f9bf1db229b4fd77
Reviewed-on: https://chromium-review.googlesource.com/951963
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
When hooks multiple cipd packages under same directory, gclient would
pop up error that same directory name appears multiple times in deps
section. This CL overrides verify_validity for CipdDependency to
support multiple packages hooked in same directory.
Bug:812386
Change-Id: Ia4f1fe0e3c8481c9b06c1d22b6e98d98e1e4c309
Reviewed-on: https://chromium-review.googlesource.com/920686
Commit-Queue: Shenghua Zhang <shenghuazhang@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Previously, it counted the number of UTF-8 bytes when spacing out the
table, not the number of code points.
Bug: 808905
Change-Id: Ice5504089e0f7097e108c6dfbbb810620b9dfc94
Reviewed-on: https://chromium-review.googlesource.com/901142
Commit-Queue: Matt Giuca <mgiuca@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Although we want to prevent dfgs from untar'ing files to a parent
or sibling of its target directory, normal files that just happen
to have ".." in their name (i.e. not preceding a path separator) are
okay.
R=hinoka
Bug: 807286
Change-Id: Ibdc2c3615c4778ef66abceb532a4f671fbdab8ef
Reviewed-on: https://chromium-review.googlesource.com/912430
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Previously, if you used custom metadata keys in a test commit (like
GitRepo.AUTHOR_NAME), they would also be treated as filenames, and fail
because they aren't strings. (This wasn't a problem because it isn't
currently used in any tests.)
Bug: 808941
Change-Id: Id7c4fa5822741925beba591ea587bd8ebbf2e478
Reviewed-on: https://chromium-review.googlesource.com/901122
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
The purpose of this CL is to remove confusion around whether or
not git-cl support the similarity flags when talking to Gerrit.
It does not. However, because Rietveld support has not been
fully ripped out of git-cl, this change does modify some of the
Rietveld-specific code to remove similarity support from everywhere.
Bug: 800902
Change-Id: I8703ed60d6889c7a36400b8a9f8f26de4669020a
Reviewed-on: https://chromium-review.googlesource.com/912389
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Also, some lines went over 80chars, so I ran 'yapf --style chromium' which
formatted some unrelated things.
BUG=905285
R=dpranke
Change-Id: Iee5f46d88a6e9782612cc4f9e5a2cb72d62ab6af
Reviewed-on: https://chromium-review.googlesource.com/907736
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
The first call to gsutil causes a network call and makes the download
script slow also in the most optimal cases. This CL refactors the
download script and moves the first gsutil call after checking locally
if sha1s match.
1) This turns the input acquisition into a generator and buffers the
files and sha1s in a list before multithreading.
2) This sequentially checks the sha1s and files and bails out early if
all match. In Chrome-land, we usually call this script with only one
file. There are some cases with around 4. This could also be
parallelized if the need arises.
3) The initial gsutil check, which ensures gsutil is updated, is moved
right in front of the multithreaded downloads.
The performance of one call to download_from_google_storage for an
existing 500MB file is 2.3s before this CL and 1.2s after (most of the
remaining time left is spent for making sha1sum).
Example for full gclient runhooks (when everything is up-to-date):
Chromium: 32s before, 12s after
V8: 12s before, 3s after
Bug: 776311
Change-Id: Ia7715a6af84b1b336455ea88494d399bdb050317
Reviewed-on: https://chromium-review.googlesource.com/897562
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Flattening currently ignores custom_vars, but it looks like that was
just an attempt to avoid including all the runtime-calculated values
(e.g. the host OS) in the flattened output, and not because there's any
real issue with supporting explicit overrides.
This is useful for changing default values in the flattened buildspecs
(e.g. to enable internal checkouts by default).
R=dpranke@google.com, iannucci@google.com
Bug: 570091,807318
Change-Id: I6d743f33cf7e3e202d5e68d21657f74f0e4c001b
Reviewed-on: https://chromium-review.googlesource.com/895927
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
This will allow us to show the progress when cloning a new repo when running
commands like gclient sync.
R=agable@chromium.org
Bug: 722686
Change-Id: I268cba343ea4c3c024292c9341d5009aa112b184
Reviewed-on: https://chromium-review.googlesource.com/890524
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
I don't know of any use-case where someone would want to
use git-cl-patch to pull a (e.g.) chromium change into
their (e.g.) v8 checkout.
Bug: 803918
Change-Id: Id53f1cc3ab97e623d0098bb366a573b18b54e91a
Reviewed-on: https://chromium-review.googlesource.com/876930
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Edward Lesmes
Michael Moss
Egor Pasko
Robert Iannucci
Benjamin Pastene
Lei Zhang
Corentin Wallez
John Budorick
Aaron Gable
Shenghua Zhang
Nodir Turakulov
Matt Giuca
Andrii Shyshkalov
Tom Anderson
Michael Achenbach