Commit Graph

82 Commits (911ccd864c77a8d9c91066858dfbd4517f9e83e2)

Author SHA1 Message Date
Gavin Mak 7f5b53ff8d Remove last py2 workarounds
python3 is the only supported version of python in depot_tools.

Changes include:
  * updating python version requirements
  * removing third_party/six

Bug: 1475402
Change-Id: Ib86231413fcac494ceb27d1783e53b9d86533c26
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4845292
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak 65c49b18b9 Drop py2 support in gclient and related files
python3 is the only supported version of python in depot_tools.

Bug: 1475402
Change-Id: I17174d7252b5cbf698700333d3cd561c6591d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4809190
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Aravind Vasudevan b6eaed26fc Add git_dependencies flag to DEPS
This change adds `git_dependencies` flag to the DEPS. This will be used to track the submodule migration state.

Design Doc: go/depot-tools-on-submodules
Change-Id: I547ad8ebbce2535960d4b6e00c0b14c00118f544
Bug: 1429149
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4591777
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
2 years ago
Bruce Dawson 03af44a516 Fix roll-dep commit message on Windows and add suggestions
roll-dep would confidently print "Commit message:" but then on Windows
would only actually give the first line of the message to git. This is
because multi-line command-lines don't actually work on all shells. This
change passes the commit message using a temporary file so that the full
message is retained on all operating systems.

This change also teaches roll-dep to give suggestions when a specified
dependency is not quite correct. This is particularly helpful if a
leading or trailing directory name is used. For instance, this command
seems plausible:

  roll-dep third_party/openh264

But in fact it is wrong and a new user has to realize that a src prefix
is needed (in general) and in this specific case a src suffix is needed
as well. Prior to this change the error message would be:

    KeyError: 'Could not find any dependency called third_party/openh264.'

But after this message it will instead say:

    KeyError: 'Could not find any dependency called third_party/openh264. Did you mean src/third_party/openh264/src'

Past me wishes I'd done this years ago.

Change-Id: I6e0d6c703906b1c1ec947788fa259bae7b7520cf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4120534
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Yuki Shiino a042264949 Allow 'noparent' in DEPS
In case of PartitionAlloc project in Chromium, there is a case that
it's useful to declare 'noparent' in the DEPS due to the situation
that a sub directory is exported to a standalone git project.

//base/allocator/partition_allocator/ (sub directory) is exported to
https://chromium.googlesource.com/chromium/src/base/allocator/partition_allocator.git

Change-Id: Ic51b6cd3928002962f25620b4d8797bed7802d1e
Bug: chromium:1151236
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3802719
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Yuki Shiino <yukishiino@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
3 years ago
Aravind Vasudevan c5f0cbb865 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: I4b8f8fc521386419a3121bbb07edc8ac83170a94
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413679
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 42c5bbbc96 Revert "Use pylint 2.7 for depot_tools"
This reverts commit 22bf605bb6.

Reason for revert: breaks gclient sync

Original change's description:
> Use pylint 2.7 for depot_tools
>
> This includes a few fixes for specific errors, and disables several new
> warnings introduced in this version, in order to allow for an incremental migration.
>
> Bug:1262286
> Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>

Bug: 1262286
Change-Id: Ieb946073c7886c7bf056ce843a5a48e82becf7a5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3413672
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Aravind Vasudevan 22bf605bb6 Use pylint 2.7 for depot_tools
This includes a few fixes for specific errors, and disables several new
warnings introduced in this version, in order to allow for an incremental migration.

Bug:1262286
Change-Id: Ie97d686748c9c952e87718a65f401c5f6f80a5c9
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3400616
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Michael Spang 0e99b9be0a Revert "Remove use_relative_hooks now that all deprecated uses are removed."
This reverts commit 4de6cfa5a9.

Reason for revert: Breaks syncing old versions.

Original change's description:
> Remove use_relative_hooks now that all deprecated uses are removed.
> 
> Bug: chromium:1107325
> Change-Id: Iac1ff8ef9426c8f57b6e12783de0ffbb0625ef2b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2349170
> Reviewed-by: Josip Sokcevic <sokcevic@google.com>
> Commit-Queue: Corentin Wallez <cwallez@chromium.org>

TBR=cwallez@chromium.org,ehmaldonado@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com,sokcevic@google.com

Change-Id: I91229c64486fe328341d1951a0b8d4dc33ee8810
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:1107325
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2352090
Auto-Submit: Michael Spang <spang@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
5 years ago
Corentin Wallez 4de6cfa5a9 Remove use_relative_hooks now that all deprecated uses are removed.
Bug: chromium:1107325
Change-Id: Iac1ff8ef9426c8f57b6e12783de0ffbb0625ef2b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2349170
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
5 years ago
Dirk Pranke fdd2cd6e5f Reland "Add a Str() function to gclient for use in DEPS files."
This relands c7eed83 with a fix to the way variables are
propagated from parent dependencies into child dependencies.

The original CL description from c7eed83 was:
> gclient's existing functionality for handling variables is
> ambiguous: the value of a variable can either be a string literal
> or an expression fragment. The implementation is required to
> parse a value as an expression, and, if it is legal, treat it
> as an expression instead of a literal. This means that
>
>   gclient_gn_args_file = 'src/build/args.gni'
>   gclient_gn_args = ['xcode_version']
>   vars = {
>     'xcode_version': 'xcode-12'
>   }
>
> would cause a problem because gclient would try to parse the
> variable as an expression, and 'xcode' would not be defined.
>
> This patch adds a workaround for this, where you can instead
> use the Str() function to explicitly tell gclient to treat the
> value as a string and not a potential expression.
>
> The above example would be changed to:
>
>   gclient_gn_args_file = 'src/build/args.gni'
>   gclient_gn_args = ['xcode_version']
>   vars = {
>     'xcode_version': Str('xcode-12')
>   }
>
> The variable may still be used in every context where it was legal
> to be used before.
>
This reverts commit 84431987dd384c79c84515004d19db67345a1c00.

Bug: 1099242
TBR=ehmaldonado@chromium.org

Change-Id: I047b871df47c367c1f34a3985e5813504e3c5c6f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2274152
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
5 years ago
Dirk Pranke ac93e6d567 Revert "Add a Str() function to gclient for use in DEPS files."
This reverts commit c7eed83f96.

Reason for revert: I'm getting reports of internal iOS checkouts being broken. Reverting while I reproduce / debug it.

Original change's description:
> Add a Str() function to gclient for use in DEPS files.
> 
> gclient's existing functionality for handling variables is
> ambiguous: the value of a variable can either be a string literal
> or an expression fragment. The implementation is required to
> parse a value as an expression, and, if it is legal, treat it
> as an expression instead of a literal. This means that
> 
>   gclient_gn_args_file = 'src/build/args.gni'
>   gclient_gn_args = ['xcode_version']
>   vars = {
>     'xcode_version': 'xcode-12'
>   }
> 
> would cause a problem because gclient would try to parse the
> variable as an expression, and 'xcode' would not be defined.
> 
> This patch adds a workaround for this, where you can instead
> use the Str() function to explicitly tell gclient to treat the
> value as a string and not a potential expression.
> 
> The above example would be changed to:
> 
>   gclient_gn_args_file = 'src/build/args.gni'
>   gclient_gn_args = ['xcode_version']
>   vars = {
>     'xcode_version': Str('xcode-12')
>   }
> 
> The variable may still be used in every context where it was legal
> to be used before.
> 
> Bug: 1099242
> 
> Change-Id: Ic2a17eea5f7098113bdba0557fe29e1a931a74b8
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2268406
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Dirk Pranke <dpranke@google.com>

TBR=thakis@chromium.org,dpranke@google.com,ehmaldonado@chromium.org,bpastene@chromium.org,apolito@google.com,infra-scoped@luci-project-accounts.iam.gserviceaccount.com

Change-Id: Iac2b003f32acdbca15a19f821b61423e34b3466c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1099242
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2273978
Reviewed-by: Dirk Pranke <dpranke@google.com>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Dirk Pranke c7eed83f96 Add a Str() function to gclient for use in DEPS files.
gclient's existing functionality for handling variables is
ambiguous: the value of a variable can either be a string literal
or an expression fragment. The implementation is required to
parse a value as an expression, and, if it is legal, treat it
as an expression instead of a literal. This means that

  gclient_gn_args_file = 'src/build/args.gni'
  gclient_gn_args = ['xcode_version']
  vars = {
    'xcode_version': 'xcode-12'
  }

would cause a problem because gclient would try to parse the
variable as an expression, and 'xcode' would not be defined.

This patch adds a workaround for this, where you can instead
use the Str() function to explicitly tell gclient to treat the
value as a string and not a potential expression.

The above example would be changed to:

  gclient_gn_args_file = 'src/build/args.gni'
  gclient_gn_args = ['xcode_version']
  vars = {
    'xcode_version': Str('xcode-12')
  }

The variable may still be used in every context where it was legal
to be used before.

Bug: 1099242

Change-Id: Ic2a17eea5f7098113bdba0557fe29e1a931a74b8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2268406
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
5 years ago
Ayu Ishii 0985861970 CodeInclusion: Rename whitelist/blacklist -> allowlist/blocklist
Migrate to more inclusive terminology.

Bug: 1097674
Change-Id: I387b385ff36c7766682c06af34ed5fc6115119d1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2268403
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Ayu Ishii <ayui@chromium.org>
5 years ago
Quinten Yearsley 925cedba7b Run spellchecker on depot_tools
This was made by running `codespell` and `scspell`
and then checking the results.

Change-Id: I169fd5b40294f83015075b4a899fbca263821f25
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2144602
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Auto-Submit: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Josip Sokcevic b3e593c1ba Escape AST string before writing to tokens
This patch enables more unit tests on Windows.

R=ehmaldonado@chromium.org

Change-Id: I8667777fc6eef11568aa2aead9850f7e37757e58
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2123808
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
5 years ago
Edward Lemur 3c11794956 gclient_eval: Raise an error on implicitly concatenated strings.
Implicitly concatenated strings are not supported when setting new
revisions, so raise an error when attempting to do se.

Bug: 1060772
Change-Id: I74d39b9eb8dcc420e9b7787c90bee642beb783cc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2099407
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur c00ac8de89 gclient: Don't allow duplicate entries on deps
Bug: 809671
Change-Id: I74103304651417e6b6889cb713fe494c95279d69
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2088422
Reviewed-by: Anthony Polito <apolito@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Edward Lemur 67cabcd71f gclient: Remove syntax validation flags.
Change-Id: I7215335eb4592cba80d31595b3bac75f55372dee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2083589
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Raul Tambre 6693d0929f gclient_eval, gclient_utils: Fix Python 3.9 removal of collections members
Various collections module members were removed in Python 3.9 after being deprecated and moved to collections.abc in earlier Python 3 versions.
For accessing these members alias collections_abc as:
* collections.abc on Python 3
* collections on Python 2

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 11, in <module>
    import gclient_utils
  File "C:\Google\depot_tools\gclient_utils.py", line 1201, in <module>
    class FrozenDict(collections.Mapping):
AttributeError: module 'collections' has no attribute 'Mapping'

Traceback (most recent call last):
  File "C:\Google\depot_tools\gclient.py", line 107, in <module>
    import gclient_eval
  File "C:\Google\depot_tools\gclient_eval.py", line 25, in <module>
    class _NodeDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

Bug: 984182
Change-Id: I7a4417978b93e29397e63764e4570a598c075bc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2043879
Auto-Submit: Raul Tambre <raul@tambre.ee>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Marc-Antoine Ruel 8e57b4bc55 python3 improvements
Ran:
  vi $(git grep --name-only iteritems | grep -v third_party)
  vi $(git grep --name-only itervalues | grep -v third_party)
  vi $(git grep --name-only 'print ' | grep -v third_party)

and edited the files quickly with adhoc macros. Then ran in recipes/:
  ./recipes.py test train

There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.

Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
5 years ago
Anthony Polito 2ae039a065 Add support for multi argument ands and ors, and the same precedence with
left to right grouping

Bug: 905740
Change-Id: I6a0b971556ab66bec841004fbbe8760b9136f216
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1793775
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Anthony Polito <apolito@google.com>
6 years ago
Edward Lemur bfcde3c5b8 gclient_eval: Make compatible with Python 3.
A new-line token is being added on Python 3, so add it explicitly
so that it also appears on Python 2.

Bug: 984182
Change-Id: I3bfb6ca83d94b6a3452b938f35f5a3174fe85903
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1764465
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
James Darpinian f994d87e56 Revert "Reland "depot_tools: Move six to depot_tools' .vpython""
This reverts commit d52b306f84.

Reason for revert: Broke ANGLE CQ with "ImportError: No module named six"
Example: https://ci.chromium.org/p/angle/builders/try/mac-dbg/6758

Original change's description:
> Reland "depot_tools: Move six to depot_tools' .vpython"
> 
> This is a reland of 3a98df0bdd
> 
> Original change's description:
> > depot_tools: Move six to depot_tools' .vpython
> > 
> > Bug: 984182
> > Change-Id: I34a242c8607624beb0f06a27c563e413dec99790
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1717492
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> > Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> > Reviewed-by: Michael Moss <mmoss@chromium.org>
> 
> Bug: 984182
> Change-Id: Ibf7034201e65a18f833cf1e7432f81a903a99aae
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1732880
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=iannucci@chromium.org,mmoss@chromium.org,tandrii@chromium.org,ehmaldonado@chromium.org

Change-Id: Id4488249b607441df9f00aa51f4a1f2bc4ad5cfd
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 984182
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1739582
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
6 years ago
Edward Lemur d52b306f84 Reland "depot_tools: Move six to depot_tools' .vpython"
This is a reland of 3a98df0bdd

Original change's description:
> depot_tools: Move six to depot_tools' .vpython
> 
> Bug: 984182
> Change-Id: I34a242c8607624beb0f06a27c563e413dec99790
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1717492
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Michael Moss <mmoss@chromium.org>

Bug: 984182
Change-Id: Ibf7034201e65a18f833cf1e7432f81a903a99aae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1732880
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur ba1b1f7ab6 gclient_eval: Make compatible with Python 3.
Bug: 984182
Change-Id: I9485614658efa0ea9d50ea666210af63c3b97adc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1716131
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Ben Pastene a541b28d4e Add 'X in tuple()' support to gclient conditionals.
This is purely so I can land:
https://chromium-review.googlesource.com/c/chromium/src/+/1625812
instead of:
https://chromium-review.googlesource.com/c/chromium/src/+/1625813

That is, the former is a slightly cleaner version of the latter. However,
it doesn't work w/o this patch.

Bug: 947531
Change-Id: If8e7f8080ba5f40408680b598fed511df89be8bb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1626479
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
6 years ago
Aaron Gable ac9b0f3786 Restore py2/py3 compatibility with basestring
A previous CL replaced "basestring" with "str", because basestring
does not exist in Python 3. However, this broke Python 2's ability
to interoperate with unicode strings. This CL introduces a workaround
(defining basestring to be equivalent to string, if it doesn't exist
already), and restores the references to basestring. This workaround
can be fixed when we're 100% on Python 3.

It also undoes some unnecessary and harder-to-read formatting changes.

Bug: 942522
Change-Id: I4a31ee46dc048134c2e4832b6c44ea00ce341899
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1572441
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Raul Tambre b946b2389c Improve gclient Python 3 compatibility
This enables gclient sync and gclient runhooks to run, barring hook script failures.
git cl upload also now works.

The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 942522
Change-Id: I2ac587b5f803ba7f5bb5e412337ce049f4b1a741
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1524583
Commit-Queue: Raul Tambre <raul@tambre.ee>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Edward Lemur fb8c1a203d gclient eval: Replace monorail with crbug.com in error message.
Tbr: tandrii@chromium.org
Bug: 914712
Change-Id: Iff8aecff346692ef36fd653f226b48270fefa304
Reviewed-on: https://chromium-review.googlesource.com/c/1380972
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Edward Lemur a1e4d48a10 setdep: Add support for variables when setting CIPD vars.
Bug: 914712
Change-Id: I13db57af519bb254389bc6bd1f1f81ad1d497e42
Reviewed-on: https://chromium-review.googlesource.com/c/1377295
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
6 years ago
Edward Lemur 8f8a50d00a gclient: Make built-in vars available for expansion.
Make it possible to refer to built-in variables without having to declare
then in DEPS files.

Bug: None
Change-Id: I5403963052463befc074f29750de56cce13927ce
Reviewed-on: https://chromium-review.googlesource.com/c/1312234
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
6 years ago
Corentin Wallez a68660d0bb gclient: add use_relative_hooks
When a recursive dependency has use_relative_paths it also makes sense
to have the hooks working directory by the dependency's directory.
Otherwise if a hook uses one of the relative dependencies it is impossible
to know which path prefix to use.

However we cannot change the behavior of hooks with use_relative_paths
because it would break existing projects that use_relative_paths but
hardcoded the prefix for hooks. Instead we add a second boolean,
use_relative_hooks that triggers the behavior.

Adds tests for the new behavior and a test for existing interactio
between hooks and recursedeps.

BUG=chromium:875245

Change-Id: Ie4c526baa425ff887b3be54e0feca7c597ededec
Reviewed-on: https://chromium-review.googlesource.com/1213327
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur 5cc2afd9b8 gclient_eval: Handle non-string variables.
Bug: 877902
Change-Id: I77a1f213318d1e09ae050c2ea0113b2976bd259e
Reviewed-on: https://chromium-review.googlesource.com/1191928
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur a92b961190 gclient setdep: Add support for CIPD versions
gclient setdep used to assume that the version of a CIPD package was always
of the form 'version:<something>'.
However, the following are all valid:
  * SHA1 hashes that specify a single instance
  * tags (tagname:tagvalue)
  * References (e.g. 'latest') that are dynamically resolved by the CIPD server.

This change adds support for all of them:
  * GetCIPD simply returns whatever the 'version' entry is, without assuming
    it begins with 'version:'.
  * SetCIPD sets 'version' ta whatever it is passed, without prepending 'version:'
    at the beginning.

Bug: 858978
Change-Id: I53669c5df7fb51cde42e0af271139b5719a47622
Reviewed-on: https://chromium-review.googlesource.com/1120943
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur e05f18d477 Reland "Reland "Expand variables in gclient flattened output.""
This is a reland of ff62224443
should_process was set to None in the previous attempt, so CIPD dependencies were not processed.
This CL fixed that.

Original change's description:
> Reland "Expand variables in gclient flattened output."
>
> This is a reland of a32f98e652
>
> Original change's description:
> > Expand variables in gclient flattened output.
> >
> > Bug: 848990
> > Change-Id: I0ad7e4f965973edbc5a335bd30f9cbd7b04abff2
> > Reviewed-on: https://chromium-review.googlesource.com/1085996
> > Reviewed-by: Michael Moss <mmoss@chromium.org>
> > Reviewed-by: Aaron Gable <agable@chromium.org>
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
>
> Tbr: agable@chromium.org
> Bug: 848990
> Change-Id: I7843544b79b2ab7e2046c187d13ea3eb65fc1b7d
> Reviewed-on: https://chromium-review.googlesource.com/1085975
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Bug: 848990
Change-Id: Ic804be1b84bf8402a741a4189b60372075dfb6f3
Reviewed-on: https://chromium-review.googlesource.com/1087368
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
John Budorick b990770386 Revert "Reland "Expand variables in gclient flattened output.""
This reverts commit ff62224443.

Reason for revert: spec revert for crbug.com/849374

Original change's description:
> Reland "Expand variables in gclient flattened output."
> 
> This is a reland of a32f98e652
> 
> Original change's description:
> > Expand variables in gclient flattened output.
> >
> > Bug: 848990
> > Change-Id: I0ad7e4f965973edbc5a335bd30f9cbd7b04abff2
> > Reviewed-on: https://chromium-review.googlesource.com/1085996
> > Reviewed-by: Michael Moss <mmoss@chromium.org>
> > Reviewed-by: Aaron Gable <agable@chromium.org>
> > Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> 
> Tbr: agable@chromium.org
> Bug: 848990
> Change-Id: I7843544b79b2ab7e2046c187d13ea3eb65fc1b7d
> Reviewed-on: https://chromium-review.googlesource.com/1085975
> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=agable@chromium.org,mmoss@chromium.org,ehmaldonado@chromium.org

Change-Id: I89ffb80893be72f60043e8a6e02f6c0fb6fe5cc1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 848990
Reviewed-on: https://chromium-review.googlesource.com/1086747
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
7 years ago
Edward Lemur ff62224443 Reland "Expand variables in gclient flattened output."
This is a reland of a32f98e652

Original change's description:
> Expand variables in gclient flattened output.
>
> Bug: 848990
> Change-Id: I0ad7e4f965973edbc5a335bd30f9cbd7b04abff2
> Reviewed-on: https://chromium-review.googlesource.com/1085996
> Reviewed-by: Michael Moss <mmoss@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

Tbr: agable@chromium.org
Bug: 848990
Change-Id: I7843544b79b2ab7e2046c187d13ea3eb65fc1b7d
Reviewed-on: https://chromium-review.googlesource.com/1085975
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lesmes 01cb51055e Revert "Expand variables in gclient flattened output."
This reverts commit a32f98e652.

Reason for revert:
Doesn't work when there are variables which definition includes other variables, e.g.:

'cros_download_vm': '"{cros_board}" == "amd64_generic"',

Original change's description:
> Expand variables in gclient flattened output.
> 
> Bug: 848990
> Change-Id: I0ad7e4f965973edbc5a335bd30f9cbd7b04abff2
> Reviewed-on: https://chromium-review.googlesource.com/1085996
> Reviewed-by: Michael Moss <mmoss@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

TBR=agable@chromium.org,mmoss@chromium.org,ehmaldonado@chromium.org

Change-Id: I7e81000e92aa352e45b420845bb1dcc8572f1962
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 848990
Reviewed-on: https://chromium-review.googlesource.com/1085974
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur a32f98e652 Expand variables in gclient flattened output.
Bug: 848990
Change-Id: I0ad7e4f965973edbc5a335bd30f9cbd7b04abff2
Reviewed-on: https://chromium-review.googlesource.com/1085996
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lemur 16f4bad914 gclient: Merge hook_os with hooks and deps_os with os.
This is done in gclient_eval, so we can remove all code
that deals with deps_os and hooks_os from gclient.

Bug: 839925
Change-Id: I491819207a712d62008ff010e313add87d22c937
Reviewed-on: https://chromium-review.googlesource.com/1058375
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Michael Moss 848c86e3e8 Make 'gclient_gn_args*' handling consistent between sync and flatten.
Previously, 'gclient sync' would process "gn_args" settings in any
recursed DEPS files, potentially producing multiple output files or
conflicting output files, but 'gclient flatten' would only ever include
one set of "gn_args" settings in the flattened output, and only if they
occurred in the top-level DEPS file.

This makes 'gclient sync' and 'gclient flatten' more consistent by
restricting them both to a single instance of those setting, and
requiring those setting to be defined either in the top-level DEPS file,
or in a recursedeps file specificed by the new 'gclient_gn_args_from'
setting.

R=dpranke@google.com, ehmaldonado@google.com

Bug: 825063
Change-Id: If90d952e47367c50b36daade16a26b29aec0c9db
Reviewed-on: https://chromium-review.googlesource.com/1039870
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Michael Moss <mmoss@chromium.org>
7 years ago
Edward Lesmes 411041f9be gclient: Add a command to get revision/variable information.
Introduce a getdep command to get revision/version/variable value information
as a counterpart for gclient setdep.

It will be useful for autorollers.

Bug: 760633
Change-Id: Iabeae0e78c6fbdcb1a3a79cfb380ac2d83f256d5
Reviewed-on: https://chromium-review.googlesource.com/999171
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lesmes 1118a2193b gclient_eval: Support unpinned dependencies in SetRevision.
If a dependency was unpinned, support adding a revision to it.

R=agable@chromium.org

Bug: 760633
Change-Id: Id2c9fe5174458acaea334726176b88558425ef5a
Reviewed-on: https://chromium-review.googlesource.com/998735
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
7 years ago
Edward Lesmes 8d6265766b gclient_eval: Add more support when adding new variables.
Now we respect comments before the first variable.

Bug: 760633
Change-Id: Ibe60d719429c033415bfb1c99942c9d04601d967
Reviewed-on: https://chromium-review.googlesource.com/998683
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Edward Lesmes 3d99381342 gclient setdep: Add support to create new variables.
If the variable doesn't already exist, gclient setdep will attempt to
create a new variable with the given name and value as the first variable
in the vars dict.

R=agable@chromium.org, mmoss@chromium.org

Bug: 760633
Change-Id: I2462e70c3695a730f87c58e56c639104efbfa54a
Reviewed-on: https://chromium-review.googlesource.com/989282
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Michael Moss 012013ee59 Add tests and fixes for |None| url values in deps entries.
This is useful for processing purely "local" DEPS files, with a command-line like:
  gclient revinfo --output-json /tmp/out \
    --spec 'solutions=[{"name": ".", "deps_file": "myDEPS", "url": None}]'

This is specifically to fix the "revinfo --output-json" case, but it
also cleans up some errors in the 'flattening' test cases. Note that
None values already work when using 'revinfo' with stdout output, it's
just the json output that complains.

BUG=825063

Change-Id: If5f69ba51cb9d0aa0f2b48a2f9b4ed8706b4c738
Reviewed-on: https://chromium-review.googlesource.com/989022
Commit-Queue: Michael Moss <mmoss@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Edward Lesmes 62af4e40a8 gclient setdep: Add support for more use cases
Add support for more ways to express revisions in DEPS files, notably
"<origin-url>@<revision>" which is needed for src-internal.

Bug: 760633
Change-Id: I86724b9c077c6101a0c424d80460123d15533da8
Reviewed-on: https://chromium-review.googlesource.com/988493
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago
Edward Lesmes 6c24d37fe9 Reland "gclient eval: Expand vars while parsing DEPS files"
This is a reland of 88f9c40e0c

We no longer need the vars dict to be declared before vars can be used.
This was causing problems because gclient flatten printed vars after deps,
breaking the above assumption.

Original change's description:
> gclient eval: Expand vars while parsing DEPS files
>
> Introduce a Parse function that takes care of expanding vars while parsing
> the DEPS file.
>
> It wraps Exec and exec calls, and supports deferring the expansion until
> later, so gclient flatten gets access to the unexpanded version.
>
> Bug: 821199
> Change-Id: I943b021cc4474c9cda67b3816b841dd8ada3f5b2
> Reviewed-on: https://chromium-review.googlesource.com/973749
> Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
> Reviewed-by: Aaron Gable <agable@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>

Bug: 821199
Change-Id: I583df23558f91871e1a2aa2574c20d35e54635f6
Reviewed-on: https://chromium-review.googlesource.com/981086
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
7 years ago