Commit Graph

141 Commits (c3daf77a40e8ba8269c57294ae94e45c96cd32c7)

Author SHA1 Message Date
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Josip Sokcevic 911ccd864c [win_toolchain] Add DIR_METADATA
This should route crash reports (e.g. https://crbug.com/1451128) to the
right component.

Fixed: b/285971982
Change-Id: If3391e4fb2b7af4f9fe91261283e8ef79a9e6759
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5141486
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Auto-Submit: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
1 year ago
Gavin Mak f936d540e1 Remove __future__ imports
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py

Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak 512f3cb37e Drop py2 support in various files
python3 is the only supported version of python in depot_tools.
Remove py2 support from files including:
  * cpplint.py
  * subprocess2.py
  * many tests and testing_support files

Bug: 1475402
Change-Id: I67a98188bc13c4dc119e6158a37bd236bfd6ea70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4824474
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
2 years ago
Bruce Dawson 04206434e7 Improve .boto message visibility
An out of date .boto file can prevent Google employees from being able
to download the Chromium toolchain from Google storage. An error message
was added to gsutil.py to help developers diagnose this, but the message
was swallowed by get_toolchain_if_necessary.py. This change prints the
error message, and adds '*' * 80 separators to make it more visible.

An obsolete import is also removed.

Bug: 1414152
Change-Id: I69d5a786672304ed363c291e0a8ef89f8be0f2e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4310059
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Ho Cheung d437c34c1c [win_toolchain] Update toolchain version
Chromium has now switched to MSVS2022+SDK22621 toolchain.
We should update some parameters and annotations in the file
to avoid inconsistent situations.

After applying this patch locally, no abnormal situation
is found. We can pack the toolchain normally.

Change-Id: I10ca115a076001efaa2b020251f2e448622598dc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4297895
Auto-Submit: Ho Cheung <uioptt24@gmail.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Bruce Dawson 0115386a26 Switch from VS 2017/2019 support to VS 2019/2022
This drops support for packaging toolchains based on VS 2017 and adds
support for VS 2022. This will allow us to package a toolchain for
building Chrome using the latest tools.

Bug: 1273169
Change-Id: I1981c935d1a9f3f4ae25e4643b9956fbdd705be1
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4054349
Reviewed-by: Etienne Pierre-Doray <etiennep@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Josip Sokcevic 6a8680a80e Use python3 for cpplint and get_toolchain on Win
R=gavinmak@google.com

Change-Id: I7d9ca9530640ffa4b8e60c247f809b0ff477d88e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673716
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
David Dorwin 7d2693ad8d Remove scottmg from OWNERS
depot_tools version of https://crrev.com/c/3526975.

Change-Id: I9c46525322ebfa36ebde67bb5fe00249cdf51fca
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3591072
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: David Dorwin <ddorwin@chromium.org>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
3 years ago
Josip Sokcevic 4de5deacd4 Explicitly run everything with python3
R=aravindvasudev@google.com, gavinmak@google.com

Change-Id: Iaa5e960c6226dea3a0814c7cb778d952eafb1502
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3525960
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Chris Blume bd68291e20 Reland "Support multiple VS installations" (with fix)
This reverts commit 9ce8be3339.

Reason for revert: To apply fix

Original change's description:
> Revert "Support multiple VS installations"
>
> This reverts commit 36d41ceff8.
>
> Reason for revert: Script references VS_VERSION variable that was
> renamed in the prior change, so it doesn't work.
>
> Original change's description:
> > Support multiple VS installations
> >
> > Currently, package_from_installed.py assumes only one version of VS is
> > installed. It takes the path of the first installation.
> >
> > This could be incorrect in several ways:
> > - Maybe both 2017 and 2019 (the supported versions) are installed and
> >   although the user specified using 2019, the 2017 path comes first.
> > - Maybe 2019 and 2022 are installed, and the 2022 path is used even
> >   though it isn't supported.
> >
> > This CL fixes that issue by parsing the vswhere.exe output to confirm
> > the VS version matches what the user specified, using its corresponding
> > path.
> >
> > Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
> > Reviewed-by: Gavin Mak <gavinmak@google.com>
> > Reviewed-by: Chris Blume <cblume@chromium.org>
> > Commit-Queue: Chris Blume <cblume@chromium.org>
> > Auto-Submit: Chris Blume <cblume@chromium.org>
>
> Change-Id: I3d9147a7786f7f54f861087d16967b75d4afe2c5
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3504193
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

Change-Id: Ica90cb8d5ce08b8b127da64969401cb40d4aee63
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3497899
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume fba3a30684 Only call GetVSPath() once
Currently, GetVSPath() is called twice. The result if that call is used
locally in both situations.

This commit makes only one call, passing the result to where it is
needed.

Change-Id: I186bd7f0387091ad16c2791de99d1de8a89f266c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3510528
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Bruce Dawson 9ce8be3339 Revert "Support multiple VS installations"
This reverts commit 36d41ceff8.

Reason for revert: Script references VS_VERSION variable that was
renamed in the prior change, so it doesn't work.

Original change's description:
> Support multiple VS installations
>
> Currently, package_from_installed.py assumes only one version of VS is
> installed. It takes the path of the first installation.
>
> This could be incorrect in several ways:
> - Maybe both 2017 and 2019 (the supported versions) are installed and
>   although the user specified using 2019, the 2017 path comes first.
> - Maybe 2019 and 2022 are installed, and the 2022 path is used even
>   though it isn't supported.
>
> This CL fixes that issue by parsing the vswhere.exe output to confirm
> the VS version matches what the user specified, using its corresponding
> path.
>
> Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
> Reviewed-by: Gavin Mak <gavinmak@google.com>
> Reviewed-by: Chris Blume <cblume@chromium.org>
> Commit-Queue: Chris Blume <cblume@chromium.org>
> Auto-Submit: Chris Blume <cblume@chromium.org>

Change-Id: I3d9147a7786f7f54f861087d16967b75d4afe2c5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3504193
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume 36d41ceff8 Support multiple VS installations
Currently, package_from_installed.py assumes only one version of VS is
installed. It takes the path of the first installation.

This could be incorrect in several ways:
- Maybe both 2017 and 2019 (the supported versions) are installed and
  although the user specified using 2019, the 2017 path comes first.
- Maybe 2019 and 2022 are installed, and the 2022 path is used even
  though it isn't supported.

This CL fixes that issue by parsing the vswhere.exe output to confirm
the VS version matches what the user specified, using its corresponding
path.

Change-Id: I2029a4f7126d0a45b5370ad58ab257df55571b3b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3458722
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Chris Blume <cblume@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume 0cfa90e203 Adjust variable names to match style guide
Currently, several non-consts are named as if they are const. The style
guide has a naming convention that fits these more accurately.

This CL changes the names to match the style guide.

Change-Id: Ib3425b76208e853b9049abe0e1be40e179124c57
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3472135
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Chris Blume d6d7a05085 Replace hard-coded '\\' with os.path.sep
Currently, the Windows-specific path separated is hard-coded in many locations.

This CL replaces several of them with os.path.sep to be less fragile.

Change-Id: Ib8ffca5743254949808a6b67e4915f8f01e044e8
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3470977
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume b3c6512be6 Correct trailing path separator
In [1], I landed a change that contained a bug. It meant to trim off a
trailing path separator. Instead of trimming the end, it trimmed
everything but the first char (since the path separator is 1 char long).

This CL fixes that bug.

[1] https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3451982

Change-Id: Ie4aebfc1042908437bd603da1c20681f19c50f73
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3473285
Auto-Submit: Chris Blume <cblume@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
3 years ago
Chris Blume 1ba4135766 Replace hard-coded Windows SDK path
Currently, the Windows SDK path is hard-coded. This does not work if the
SDK is installed in a different path or drive.

This CL fetches the actual Windows SDK path and uses it instead.

Change-Id: I31808ba0dbe76f47253fc165ccbd8f027bf396ec
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3451982
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Chris Blume <cblume@chromium.org>
Auto-Submit: Chris Blume <cblume@chromium.org>
3 years ago
Viktor Samun 0aabd63461 Fix python2/3 compatibility in DownloadUsingHttp function
Change-Id: I59ed965d066b7875c1739408054498b328652be5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988315
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
4 years ago
Dirk Pranke 1fc595b55c Fix py3 compat issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I6ee0cb6be8e86490ec0ecf5a4f2e236546bb92ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2900993
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
4 years ago
Robert Liao f5c054ce53 Pretty-Print the win_toolchain JSON Output
This makes the file easier to read.

Change-Id: Ie5eac66582bd6f3ce3c31def6f591e001de9a79a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893667
Auto-Submit: Robert Liao <robliao@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Dirk Pranke 3b9d2d8557 Fix a Python3-compatibility issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I9205089c3a7d9f2d3e2eb9aac7ef6da9ae9b0876
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2890330
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 3f562c04e0 win_toolchain: Write json files only if different from what is on disk
Bug: chromium:1188672
Change-Id: I10aed281afd636acc0c680a7da486ae74aa637df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2807968
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 399c5918bf win: minor behavior-preserving tweaks to get_toolchain_if_necessary.py
- add run line (also to package_from_installed.py while here)
- use startswith() to check if a string starts with another
- switch from optparse to argparse
  - ...and make desired-hash a required positional argument
- drop win8 sdk support (not behavior preserving, but also unused
  for years)

Bug: none
Change-Id: I73056184208a48b9d9610f330c56f3a324763195
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2679295
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 3fe941c45e win toolchain packager: Put win sdk in "Windows Kits/10/" instead of "win_sdk/"
This is so that the packaged toolchain can be used with clang-cl's
new /winsysroot flag added in https://reviews.llvm.org/rG82847436e.
No impact yet on the chrome build yet -- the tooling should
transparently get the new Windows Sdk path via the generated SetEnv
json file and use it with -imsvc. (I tested this locally by tweaking
my installed hermetic win package to look like the one generated
by this CL.)

Once this is deployed, we can switch the chrome build to use
/winsysroot if we want to -- it'd make compile command lines
a bit shorter and easier to work with.

Depends on https://chromium-review.googlesource.com/c/chromium/src/+/2665866

Bug: 1173176
Change-Id: I04c435f2323f26e3c26ed82656929809a7e0b5e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2655836
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 474cd16266 win toolchain: The SDK path should be checked per toolchain dir
This is a follow-up to crev.com/c/2668127. That patch incorrectly
checked the windows sdk path style for the target toolchain, but
there might be other, older toolchains with the other path style.
We need to check this per toolchain directory.

Also make sure ignored_dirs is compared case-insensitively.

Bug: 1173176
Change-Id: I005eb1b3200b11597978936a970f50f101708bea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2669048
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 499a2fd61b Reland "win toolchain: Prepare downloader for windows sdk dir switch"
This is a reland of e72789f5b4

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

Bug: 1173176,1173393
Change-Id: Ic706f694f8f0260208fa637864e62d7cc4f7ce93
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2668127
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Takuto Ikuta 98cc028bbe Revert "win toolchain: Prepare downloader for windows sdk dir switch"
This reverts commit e72789f5b4.

Reason for revert: 
This broke goma builder.

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

TBR=thakis@chromium.org,brucedawson@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com

Change-Id: I8d133f4fa199f81978f5245bdb2155c62bc9cc88
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1173176
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666651
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Nico Weber e72789f5b4 win toolchain: Prepare downloader for windows sdk dir switch
crrev.com/c/2655836 tries to move the Windows SDK from
"win_sdk" to "Windows Kits/10".

get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
third_party/depot_tools/win_toolchain/data.json which then gets copied
by a script in the chromium repo to build/win_toolchain.json.
For the SDK move to work, chromium's pinned depot_tools
must write the new SDK path when rolling in the new toolchain package.
This change makes depot_tools handle win packages that have the
windows sdk either win "win_sdk" or in "Windows Kits\10".

The plan is:

1. Land this change, which can handle both path styles
2. Wait for depot_tools in chromium to update
3. Then roll to a win toolchain package with the new layout

In a few years, when we no longer need the old layout,
we can remove this detection code again and assume the new layout.

Bug: 1173176
Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber cb07c525c0 win toolchain: Remove some old MSVS 2013 references.
We used to use this many years ago.

Bug: none
Change-Id: I1df268c793bbfe7423ecf86ea99ebbe918e280d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2658676
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Josip Sokcevic 9c0dc3026e Replace non-inclusive terms
Bug: 1118410, 1118413, 1118422, 1118424, 1118426
Bug: 1118428, 1118430, 1118434, 1118438, 1118442
Bug: 1118453, 1118454
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: build
Change-Id: I43e6305f8c3136774f01b57f12bb442bcb863371
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2551388
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
5 years ago
Bruce Dawson 21de33a93c Print packaging file size/count stats always
When doing a dry run of packaging the toolchain we have always printed
stats on the size and number of files. It is odd and annoying that we
don't print those stats when doing the actual packaging, so this changes
that.

Change-Id: Ic60c3f4960dacaea39c048dc0c5f4c50b551f68a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2380231
Reviewed-by: Mirko Bonadei <mbonadei@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson 7872beb489 Shorten hash to avoid MAX_PATH limits
The Windows SDK contains some very long paths. When those are added to
the 40-character hash directory and the other path components it is easy
to hit the Windows 260-character MAX_PATH limit. In addition, the 40
character hash makes paths unwieldy in VsChromium search results and
elsewhere.

A ten character hash should be more than enough to avoid collisions - if
we upload a million toolchain packages then there will be about a 50%
chance of a name collision - we won't upload more than a thousand.

This was tested by copying the current toolchain file on Google storage
to a truncated name and then changing to that hash in vs_toolchain.py.
It was also necessary to copy the updates from depot_tools to
third_party\depot_tools. This means that we can't actually start using
short hashes until depot_tools has rolled.

We probably won't use a shorter hash until we next roll the toolchain,
just because changing the toolchain hash is mildly disruptive.

Bug: 1120785
Change-Id: I878b058857cbe9cb72a72b535864404eede33f3f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2376030
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson e95b5d6ad5 Avoid ..\.. in packaged environment variables
win_sdk\bin\setenv.cmd is used to configure the build environment for
the packaged toolchain. This has always been done by creating paths such
as %~dp0..\..\VC, which leads to extra long paths that are less readable
than direct paths. These extra long paths make us more vulnerable to
MAX_PATH issues.

This change generates the paths relative to the grandparent directory.
It does that by changing the current directory to the grandparent
directory of the script and then generating paths relative to %cd%.
The paths are still absolute they are just shorter and simpler.

The paths are also stored in SetEnv.*.json and this change alters the
relative-root used for these paths. crrev.com/c/2370604 detects the
needed relative-root to handle toolchains packaged before and after this
change.

This change was tested using crrev.com/c/2370604 (temporarily modified
to use a toolchain created by this change) to ensure that it works.

Bug: 1120785
Change-Id: Icd72e36766d7ec92a491d16dbfd9ad7fc1b2ebc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2372727
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson f0e672e732 Filter out Assessment and Deployment Kit
When packaging the VC++ and Windows SDK toolchain it is important to
filter out unneeded directories both to avoid unnecessary size and to
avoid including excessively long paths. The Assessment and Deployment
Kit was installed on my test machine, is 3.5 GB, and has some very
long file paths, so now I filter it out.

Bug: 1120785
Change-Id: I9e0c16cea57cf4a687db1e4c315440fd5a50c9f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2370864
Reviewed-by: Xiaohan Wang <xhwang@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson 355e6449e7 Python 3 support for get_toolchain_if_necessary.py
get_toolchain_if_necessary.py would fail on Python 3 when passing a
string to digest.update - this string needs to be encoded to bytes.
This even stops packaging of new toolchains from working.

With this change digest.update get_toolchain_if_necessary.py works with
Python 2 and Python 3.

As a test the current toolchain directory was repackaged - the same hash
was generated.

Change-Id: Ia682d15be42521a35f4df2e14d1f715d9e42d96f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2234586
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson e65444f91d Fix Python3 bugs in package_from_installed.py
The package_from_installed.py script didn't run under Python 3. This
change fixes that and this script now _requires_ Python 3. Requiring
Python 3 was simpler than supporting both, and there is no need to
maintain Python 2.7 support. This script is only ever invoked manually
and all machines it is run on should have Python 3.

Changes included using universal_newlines=True so that
subprocess.check_output returns a string instead of bytes, and
by explicitly converting dict.items() to a list.

Similarly, open(name, 'wb') was changed to open(name,'wt', newline='')
to avoid byte/str problems while still avoiding CR/LF translations.

This also changes the default Windows 10 SDK version to something more
recent - the current version that we build with.

To test:

  python3 package_from_installed.py -d 2019

The use of '-d' makes the script run in test mode which is much faster.

Change-Id: I2e3fc4d464f2914ba5ae36f0a78570ea7e5d4f03
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2224506
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Dale Curtis 0e7788d16b Skip sources downloaded by WinDbg when checking toolchain status.
This avoids the toolchain being wiped after using WinDbg.

Bug: None
Change-Id: I989ef7744b46254bab4c5a707724dc38a9b9a548
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2159998
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Auto-Submit: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
5 years ago
Yuly Novikov e93101895e Also set LIBPATH in Windows toolchain environment
Needed to find *.winmd files for UWP build.

This CL and crrev.com/c/2007881 were used to generate the recent
toolchains with UWP support:
b92fff97f2e0323e99803f37f4b77b843bdbf69d in crrev.com/c/2015984 and
9ff60e43ba91947baca460d0ca3b1b980c3a2c23 in crrev.com/c/2025528.

Bug: chromium:1032635
Change-Id: I1ab0b6e11a30600deb377078815c268d8f9027e4
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2013579
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Yuly Novikov 852d4aa1ba Don't drop windows.winmd when packaging Windows toolchain
It is needed for UWP build and is the same on at least 2 machines in
C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.18362.0\Windows.winmd

Bug: chromium:1032635
Change-Id: I9d7148a8dc11ea85f1b930f63ae19eaa58ac2bf6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2007881
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
5 years ago
Marc-Antoine Ruel 8e57b4bc55 python3 improvements
Ran:
  vi $(git grep --name-only iteritems | grep -v third_party)
  vi $(git grep --name-only itervalues | grep -v third_party)
  vi $(git grep --name-only 'print ' | grep -v third_party)

and edited the files quickly with adhoc macros. Then ran in recipes/:
  ./recipes.py test train

There was only a small subset of files that had been updated to use
six.iteritems() and six.itervalues(). Since the dataset size that is
being used in gclient is small (pretty much always below 200 items),
it's better to just switch to .items() right away and take the temporary
performance hit, so that we don't need to come back to rewrite the code.

Recipe-Nontrivial-Roll: build
Bug: 984182
Change-Id: I5faf11486b66b0d73c9098ab0f2ce1b15a45c53e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854900
Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Auto-Submit: Marc-Antoine Ruel <maruel@chromium.org>
6 years ago
Mirko Bonadei cbff54b4ac Fail to repackage if win_sdk\Debuggers doesn't exist.
This is easy to miss and a toolchain package without win_sdk\Debuggers
will not work as expected so this CL adds a check to stop the
packaging/repacking process as soon as possible.

Bug: 1006238
Change-Id: I272fbaf3e1cbde48a3e260e38e34ffc10d6c86df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1829272
Commit-Queue: Mirko Bonadei <mbonadei@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Mirko Bonadei e4174f483e Add VS 2019 support to package_from_installed.py.
This CL also removes VS 2015 support.

Bug: 1006238
Change-Id: Ib05b3d211341fcd5805c3acca3bc4c0aa894831d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1823961
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Mirko Bonadei <mbonadei@chromium.org>
6 years ago
Daniel Bratell f411acf552 Make it possible to build a Windows SDK package without arm libs
Not everyone builds Chromium for arm so then it's practical to be
able to use a smaller SDK package without arm support included.

This also removes some samples that should not be needed.

Recipe-Nontrivial-Roll: chromiumos
Change-Id: I10f64d8907600fe79d3051083d100a7562d3a797
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1752202
Commit-Queue: Daniel Bratell <bratell@opera.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Raul Tambre 80ee78e7fa Convert print statements to Python 3 style
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.

The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
6 years ago
Tom Anderson e893454f79 Fix a typo
BUG=None
R=dpranke

Change-Id: Ia16a3a30bc8339d81847725ed9a23b2694e4169e
Reviewed-on: https://chromium-review.googlesource.com/c/1464772
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Lei Zhang b9ae2ca9a5 Fix get_toolchain_if_necessary.py --nodownload message formatting.
And indent properly as well.

Change-Id: I78e1f67d7820120bf809f1ac3ab64ce48c74e804
Reviewed-on: https://chromium-review.googlesource.com/c/1458784
Commit-Queue: Lei Zhang <thestig@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson b19e8dff15 Add a --nodownload option to get_toolchain_if_necessary.py
The Windows toolchain should be downloaded during gclient runhooks,
however because get_toolchain_if_necessary.py can either configure an
existing toolchain or download a toolchain there have been problems when
the toolchain is downloaded at an unexpected point. In particular, a
toolchain download during gn gen can lead to parallel downloads and
configuration failures.

The --nodownload option will be passed (after a subsequent change) by
build\vs_toolchain.py. A --nodownload option is needed instead of a more
positive --allowdownload option because of backwards compatibility. This
change will have no effect until the build\vs_toolchain.py change is
landed.

Bug: 584393, 662325
Change-Id: I9a0950e066744c0310e49e45d001a9113f1831cf
Reviewed-on: https://chromium-review.googlesource.com/c/1439805
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson eba46fc12e Improve error message when a toolchain can't be found
When non-Googlers try to build Chromium for the first time they get an
error saying:

  Please follow the instructions at https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md

However those instructions are quite long and it's not obvious to some
users which part of the instructions they have forgotten. The most
likely is that they didn't set DEPOT_TOOLS_WIN_TOOLCHAIN=0, so this
changes the error message to suggest that directly.

Auto-detecting the correct state for DEPOT_TOOLS_WIN_TOOLCHAIN was
tried in crrev.com/c/1374833 but deemed impractical.

Bug: 907300, angleproject:2712
Change-Id: I45a9f59babe90bbe2137d49c555884d7bbbcf170
Reviewed-on: https://chromium-review.googlesource.com/c/1382942
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago