Commit Graph

80 Commits (da2ef92c9f15b97a75bf50ad5928e537b1651997)

Author SHA1 Message Date
Michael Ershov 3c79cfc485 Revert "Reland "Add support for GCS deps""
This reverts commit 46d5382f69.

Reason for revert: win-presubmit builder probably started failing after this CL.
https://ci.chromium.org/ui/p/chromium/builders/ci/win-presubmit

Original change's description:
> Reland "Add support for GCS deps"
>
> This is a reland of commit 3eedee7b55
>
> Fixed more imports in download_google_storage.py
>
> Original change's description:
> > Add support for GCS deps
> >
> > Also take out GCS calling logic from download_google_storage and
> > into call_google_storage.
> >
> > GCS deps look like:
> >    'src/third_party/node/linux': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-nodejs/20.11.0',
> >        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
> >        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
> >    },
> >
> >    'src/third_party/llvm-build/Release+Asserts': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-browser-clang',
> >        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
> >        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
> >    },
> >
> > Example directory for src/third_party/node/linux after gclient sync:
> > - tar_file.gz is the downloaded file from GCS.
> > - node_linux_x64/ is extracted in its path.
> > - `hash` contains the sha of GCS filename.
> > ```
> > chromium/src/ ->
> >    third_party/node/linux/ ->
> >        hash, tar_file.gz, node_linux_x64/
> > ```
> >
> > Bug: b/324418194
> > Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> > Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
>
> Bug: b/324418194
> Change-Id: Ie64265a86abcec0135408715a45c32a8bb7c7408
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5346338
> Reviewed-by: Joanna Wang <jojwang@chromium.org>
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>

Bug: b/324418194
Change-Id: Ic4517f6c9e05aea3f3f052d2a44865733236998b
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5352167
Auto-Submit: Michael Ershov <miersh@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Owners-Override: Michael Ershov <miersh@google.com>
1 year ago
Stephanie Kim 46d5382f69 Reland "Add support for GCS deps"
This is a reland of commit 3eedee7b55

Fixed more imports in download_google_storage.py

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: Ie64265a86abcec0135408715a45c32a8bb7c7408
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5346338
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
1 year ago
Stephanie Kim 93d5662c1d Revert "Reland "Add support for GCS deps""
This reverts commit d94a03f21e.

Reason for revert: PLATFORM_MAPPING import issue

Original change's description:
> Reland "Add support for GCS deps"
>
> This is a reland of commit 3eedee7b55
>
> Original change's description:
> > Add support for GCS deps
> >
> > Also take out GCS calling logic from download_google_storage and
> > into call_google_storage.
> >
> > GCS deps look like:
> >    'src/third_party/node/linux': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-nodejs/20.11.0',
> >        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
> >        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
> >    },
> >
> >    'src/third_party/llvm-build/Release+Asserts': {
> >        'dep_type': 'gcs',
> >        'condition': 'checkout_linux',
> >        'bucket': 'chromium-browser-clang',
> >        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
> >        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
> >    },
> >
> > Example directory for src/third_party/node/linux after gclient sync:
> > - tar_file.gz is the downloaded file from GCS.
> > - node_linux_x64/ is extracted in its path.
> > - `hash` contains the sha of GCS filename.
> > ```
> > chromium/src/ ->
> >    third_party/node/linux/ ->
> >        hash, tar_file.gz, node_linux_x64/
> > ```
> >
> > Bug: b/324418194
> > Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> > Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> > Reviewed-by: Joanna Wang <jojwang@chromium.org>
>
> Bug: b/324418194
> Change-Id: I8b960f6dd3534658a9ee0b7688e68eaa15df1444
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5334791
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: I28bc0ba838a1135ff882cd88b77dbb2cb4b85b82
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5345022
Reviewed-by: Joanna Wang <jojwang@chromium.org>
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
1 year ago
Stephanie Kim d94a03f21e Reland "Add support for GCS deps"
This is a reland of commit 3eedee7b55

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: I8b960f6dd3534658a9ee0b7688e68eaa15df1444
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5334791
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Stephanie Kim 064e03aa1c Revert "Add support for GCS deps"
This reverts commit 3eedee7b55.

Reason for revert: https://buganizer.corp.google.com/issues/324418194#comment7

Original change's description:
> Add support for GCS deps
>
> Also take out GCS calling logic from download_google_storage and
> into call_google_storage.
>
> GCS deps look like:
>    'src/third_party/node/linux': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-nodejs/20.11.0',
>        'object_name': '46795170ff5df9831955f163f6966abde581c8af',
>        'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
>    },
>
>    'src/third_party/llvm-build/Release+Asserts': {
>        'dep_type': 'gcs',
>        'condition': 'checkout_linux',
>        'bucket': 'chromium-browser-clang',
>        'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
>        'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
>    },
>
> Example directory for src/third_party/node/linux after gclient sync:
> - tar_file.gz is the downloaded file from GCS.
> - node_linux_x64/ is extracted in its path.
> - `hash` contains the sha of GCS filename.
> ```
> chromium/src/ ->
>    third_party/node/linux/ ->
>        hash, tar_file.gz, node_linux_x64/
> ```
>
> Bug: b/324418194
> Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
> Commit-Queue: Stephanie Kim <kimstephanie@google.com>
> Reviewed-by: Joanna Wang <jojwang@chromium.org>

Bug: b/324418194
Change-Id: Ie74df90065c1fea087b240d98005b9d9b4f44411
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5336079
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Stephanie Kim 3eedee7b55 Add support for GCS deps
Also take out GCS calling logic from download_google_storage and
into call_google_storage.

GCS deps look like:
   'src/third_party/node/linux': {
       'dep_type': 'gcs',
       'condition': 'checkout_linux',
       'bucket': 'chromium-nodejs/20.11.0',
       'object_name': '46795170ff5df9831955f163f6966abde581c8af',
       'sha256sum': '887504c37404898ca41b896f448ee6d7fc24179d8fb6a4b79d028ab7e1b7153d',
   },

   'src/third_party/llvm-build/Release+Asserts': {
       'dep_type': 'gcs',
       'condition': 'checkout_linux',
       'bucket': 'chromium-browser-clang',
       'object_name': 'Linux_x64/clang-llvmorg-18-init-17730-gf670112a-2.tar.xz',
       'sha256sum': '1e46df9b4e63c074064d75646310cb76be2f19815997a8486987189d80f991e8',
   },

Example directory for src/third_party/node/linux after gclient sync:
- tar_file.gz is the downloaded file from GCS.
- node_linux_x64/ is extracted in its path.
- `hash` contains the sha of GCS filename.
```
chromium/src/ ->
   third_party/node/linux/ ->
       hash, tar_file.gz, node_linux_x64/
```

Bug: b/324418194
Change-Id: Ibcbbff27e211f194ddb8a08494af56570a84a12b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5299722
Commit-Queue: Stephanie Kim <kimstephanie@google.com>
Reviewed-by: Joanna Wang <jojwang@chromium.org>
1 year ago
Gavin Mak edba22d4eb Fix multiline comment formatting
Many incorrectly formatted comments exist from the switch to
4 space indent: https://crrev.com/c/4836379

Bug: 1514505
Change-Id: I6366f9da812919bd35b999f18fa8a49b7a66c09b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5153633
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
1 year ago
Gavin Mak f936d540e1 Remove __future__ imports
All __future__ imports (unicode_literals, print_function) are already
mandatory in py3. Also remove an outdated py2 comment in
presubmit_canned_checks.py

Bug: 1475402
Change-Id: I27cf6a8268f6dd1081f22af782c4c29a975376ae
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4867135
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Gavin Mak <gavinmak@google.com>
2 years ago
Mike Frysinger 124bb8e53c switch to 4 space indent
Leave the recipes/ code at 2 space to match the rest of the recipes
project in other repos.

Reformatted using:
files=( $(
	git ls-tree -r --name-only HEAD | \
		grep -Ev -e '^(third_party|recipes)/' | \
		grep '\.py$';
	git grep -l '#!/usr/bin/env.*python' | grep -v '\.py$'
) )
parallel ./yapf -i -- "${files[@]}"
~/chromiumos/chromite/contrib/reflow_overlong_comments "${files[@]}"

The files that still had strings that were too long were manually
reformatted because they were easy and only a few issues.
autoninja.py
clang_format.py
download_from_google_storage.py
fix_encoding.py
gclient_utils.py
git_cache.py
git_common.py
git_map_branches.py
git_reparent_branch.py
gn.py
my_activity.py
owners_finder.py
presubmit_canned_checks.py
reclient_helper.py
reclientreport.py
roll_dep.py
rustfmt.py
siso.py
split_cl.py
subcommand.py
subprocess2.py
swift_format.py
upload_to_google_storage.py

These files still had lines (strings) that were too long, so the pylint
warnings were suppressed with a TODO.
auth.py
gclient.py
gclient_eval.py
gclient_paths.py
gclient_scm.py
gerrit_util.py
git_cl.py
presubmit_canned_checks.py
presubmit_support.py
scm.py

Change-Id: Ia6535c4f2c48d46b589ec1e791dde6c6b2ea858f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4836379
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Auto-Submit: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
2 years ago
Gavin Mak 512f3cb37e Drop py2 support in various files
python3 is the only supported version of python in depot_tools.
Remove py2 support from files including:
  * cpplint.py
  * subprocess2.py
  * many tests and testing_support files

Bug: 1475402
Change-Id: I67a98188bc13c4dc119e6158a37bd236bfd6ea70
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4824474
Commit-Queue: Gavin Mak <gavinmak@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@chromium.org>
2 years ago
Bruce Dawson 04206434e7 Improve .boto message visibility
An out of date .boto file can prevent Google employees from being able
to download the Chromium toolchain from Google storage. An error message
was added to gsutil.py to help developers diagnose this, but the message
was swallowed by get_toolchain_if_necessary.py. This change prints the
error message, and adds '*' * 80 separators to make it more visible.

An obsolete import is also removed.

Bug: 1414152
Change-Id: I69d5a786672304ed363c291e0a8ef89f8be0f2e0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/4310059
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
2 years ago
Josip Sokcevic 6a8680a80e Use python3 for cpplint and get_toolchain on Win
R=gavinmak@google.com

Change-Id: I7d9ca9530640ffa4b8e60c247f809b0ff477d88e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3673716
Auto-Submit: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Gavin Mak <gavinmak@google.com>
3 years ago
Josip Sokcevic 4de5deacd4 Explicitly run everything with python3
R=aravindvasudev@google.com, gavinmak@google.com

Change-Id: Iaa5e960c6226dea3a0814c7cb778d952eafb1502
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/3525960
Reviewed-by: Gavin Mak <gavinmak@google.com>
Reviewed-by: Aravind Vasudevan <aravindvasudev@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
3 years ago
Viktor Samun 0aabd63461 Fix python2/3 compatibility in DownloadUsingHttp function
Change-Id: I59ed965d066b7875c1739408054498b328652be5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2988315
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
4 years ago
Dirk Pranke 1fc595b55c Fix py3 compat issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I6ee0cb6be8e86490ec0ecf5a4f2e236546bb92ee
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2900993
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@google.com>
4 years ago
Robert Liao f5c054ce53 Pretty-Print the win_toolchain JSON Output
This makes the file easier to read.

Change-Id: Ie5eac66582bd6f3ce3c31def6f591e001de9a79a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2893667
Auto-Submit: Robert Liao <robliao@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Dirk Pranke 3b9d2d8557 Fix a Python3-compatibility issue in get_toolchain_if_necessary.
Bug: 1207012
Change-Id: I9205089c3a7d9f2d3e2eb9aac7ef6da9ae9b0876
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2890330
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 3f562c04e0 win_toolchain: Write json files only if different from what is on disk
Bug: chromium:1188672
Change-Id: I10aed281afd636acc0c680a7da486ae74aa637df
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2807968
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber 399c5918bf win: minor behavior-preserving tweaks to get_toolchain_if_necessary.py
- add run line (also to package_from_installed.py while here)
- use startswith() to check if a string starts with another
- switch from optparse to argparse
  - ...and make desired-hash a required positional argument
- drop win8 sdk support (not behavior preserving, but also unused
  for years)

Bug: none
Change-Id: I73056184208a48b9d9610f330c56f3a324763195
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2679295
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 474cd16266 win toolchain: The SDK path should be checked per toolchain dir
This is a follow-up to crev.com/c/2668127. That patch incorrectly
checked the windows sdk path style for the target toolchain, but
there might be other, older toolchains with the other path style.
We need to check this per toolchain directory.

Also make sure ignored_dirs is compared case-insensitively.

Bug: 1173176
Change-Id: I005eb1b3200b11597978936a970f50f101708bea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2669048
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
4 years ago
Nico Weber 499a2fd61b Reland "win toolchain: Prepare downloader for windows sdk dir switch"
This is a reland of e72789f5b4

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

Bug: 1173176,1173393
Change-Id: Ic706f694f8f0260208fa637864e62d7cc4f7ce93
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2668127
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Takuto Ikuta 98cc028bbe Revert "win toolchain: Prepare downloader for windows sdk dir switch"
This reverts commit e72789f5b4.

Reason for revert: 
This broke goma builder.

Original change's description:
> win toolchain: Prepare downloader for windows sdk dir switch
>
> crrev.com/c/2655836 tries to move the Windows SDK from
> "win_sdk" to "Windows Kits/10".
>
> get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
> third_party/depot_tools/win_toolchain/data.json which then gets copied
> by a script in the chromium repo to build/win_toolchain.json.
> For the SDK move to work, chromium's pinned depot_tools
> must write the new SDK path when rolling in the new toolchain package.
> This change makes depot_tools handle win packages that have the
> windows sdk either win "win_sdk" or in "Windows Kits\10".
>
> The plan is:
>
> 1. Land this change, which can handle both path styles
> 2. Wait for depot_tools in chromium to update
> 3. Then roll to a win toolchain package with the new layout
>
> In a few years, when we no longer need the old layout,
> we can remove this detection code again and assume the new layout.
>
> Bug: 1173176
> Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
> Commit-Queue: Nico Weber <thakis@chromium.org>
> Auto-Submit: Nico Weber <thakis@chromium.org>
> Reviewed-by: Bruce Dawson <brucedawson@chromium.org>

TBR=thakis@chromium.org,brucedawson@chromium.org,infra-scoped@luci-project-accounts.iam.gserviceaccount.com

Change-Id: I8d133f4fa199f81978f5245bdb2155c62bc9cc88
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1173176
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666651
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
4 years ago
Nico Weber e72789f5b4 win toolchain: Prepare downloader for windows sdk dir switch
crrev.com/c/2655836 tries to move the Windows SDK from
"win_sdk" to "Windows Kits/10".

get_toolchain_if_necessary.py (in depot_tools) saves the path to the SDK to
third_party/depot_tools/win_toolchain/data.json which then gets copied
by a script in the chromium repo to build/win_toolchain.json.
For the SDK move to work, chromium's pinned depot_tools
must write the new SDK path when rolling in the new toolchain package.
This change makes depot_tools handle win packages that have the
windows sdk either win "win_sdk" or in "Windows Kits\10".

The plan is:

1. Land this change, which can handle both path styles
2. Wait for depot_tools in chromium to update
3. Then roll to a win toolchain package with the new layout

In a few years, when we no longer need the old layout,
we can remove this detection code again and assume the new layout.

Bug: 1173176
Change-Id: Iaefc5c16685d3dbfff87a3e50a7b20b457366e44
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2666429
Commit-Queue: Nico Weber <thakis@chromium.org>
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Nico Weber cb07c525c0 win toolchain: Remove some old MSVS 2013 references.
We used to use this many years ago.

Bug: none
Change-Id: I1df268c793bbfe7423ecf86ea99ebbe918e280d0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2658676
Auto-Submit: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
4 years ago
Josip Sokcevic 9c0dc3026e Replace non-inclusive terms
Bug: 1118410, 1118413, 1118422, 1118424, 1118426
Bug: 1118428, 1118430, 1118434, 1118438, 1118442
Bug: 1118453, 1118454
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: build
Change-Id: I43e6305f8c3136774f01b57f12bb442bcb863371
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2551388
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
4 years ago
Bruce Dawson 7872beb489 Shorten hash to avoid MAX_PATH limits
The Windows SDK contains some very long paths. When those are added to
the 40-character hash directory and the other path components it is easy
to hit the Windows 260-character MAX_PATH limit. In addition, the 40
character hash makes paths unwieldy in VsChromium search results and
elsewhere.

A ten character hash should be more than enough to avoid collisions - if
we upload a million toolchain packages then there will be about a 50%
chance of a name collision - we won't upload more than a thousand.

This was tested by copying the current toolchain file on Google storage
to a truncated name and then changing to that hash in vs_toolchain.py.
It was also necessary to copy the updates from depot_tools to
third_party\depot_tools. This means that we can't actually start using
short hashes until depot_tools has rolled.

We probably won't use a shorter hash until we next roll the toolchain,
just because changing the toolchain hash is mildly disruptive.

Bug: 1120785
Change-Id: I878b058857cbe9cb72a72b535864404eede33f3f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2376030
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Bruce Dawson 355e6449e7 Python 3 support for get_toolchain_if_necessary.py
get_toolchain_if_necessary.py would fail on Python 3 when passing a
string to digest.update - this string needs to be encoded to bytes.
This even stops packaging of new toolchains from working.

With this change digest.update get_toolchain_if_necessary.py works with
Python 2 and Python 3.

As a test the current toolchain directory was repackaged - the same hash
was generated.

Change-Id: Ia682d15be42521a35f4df2e14d1f715d9e42d96f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2234586
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
5 years ago
Dale Curtis 0e7788d16b Skip sources downloaded by WinDbg when checking toolchain status.
This avoids the toolchain being wiped after using WinDbg.

Bug: None
Change-Id: I989ef7744b46254bab4c5a707724dc38a9b9a548
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2159998
Commit-Queue: Dale Curtis <dalecurtis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Auto-Submit: Dale Curtis <dalecurtis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
5 years ago
Raul Tambre 80ee78e7fa Convert print statements to Python 3 style
Ran "2to3 -w -n -f print ./" and manually added imports.
Ran "^\s*print " and "\s+print " to find batch/shell scripts, comments and the like with embedded code, and updated them manually.
Also manually added imports to files, which used print as a function, but were missing the import.

The scripts still work with Python 2.
There are no intended behaviour changes.

Bug: 942522
Change-Id: Id777e4d4df4adcdfdab1b18bde89f235ef491b9f
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1595684
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Raul Tambre <raul@tambre.ee>
6 years ago
Tom Anderson e893454f79 Fix a typo
BUG=None
R=dpranke

Change-Id: Ia16a3a30bc8339d81847725ed9a23b2694e4169e
Reviewed-on: https://chromium-review.googlesource.com/c/1464772
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Auto-Submit: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
6 years ago
Lei Zhang b9ae2ca9a5 Fix get_toolchain_if_necessary.py --nodownload message formatting.
And indent properly as well.

Change-Id: I78e1f67d7820120bf809f1ac3ab64ce48c74e804
Reviewed-on: https://chromium-review.googlesource.com/c/1458784
Commit-Queue: Lei Zhang <thestig@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson b19e8dff15 Add a --nodownload option to get_toolchain_if_necessary.py
The Windows toolchain should be downloaded during gclient runhooks,
however because get_toolchain_if_necessary.py can either configure an
existing toolchain or download a toolchain there have been problems when
the toolchain is downloaded at an unexpected point. In particular, a
toolchain download during gn gen can lead to parallel downloads and
configuration failures.

The --nodownload option will be passed (after a subsequent change) by
build\vs_toolchain.py. A --nodownload option is needed instead of a more
positive --allowdownload option because of backwards compatibility. This
change will have no effect until the build\vs_toolchain.py change is
landed.

Bug: 584393, 662325
Change-Id: I9a0950e066744c0310e49e45d001a9113f1831cf
Reviewed-on: https://chromium-review.googlesource.com/c/1439805
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
6 years ago
Bruce Dawson eba46fc12e Improve error message when a toolchain can't be found
When non-Googlers try to build Chromium for the first time they get an
error saying:

  Please follow the instructions at https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md

However those instructions are quite long and it's not obvious to some
users which part of the instructions they have forgotten. The most
likely is that they didn't set DEPOT_TOOLS_WIN_TOOLCHAIN=0, so this
changes the error message to suggest that directly.

Auto-detecting the correct state for DEPOT_TOOLS_WIN_TOOLCHAIN was
tried in crrev.com/c/1374833 but deemed impractical.

Bug: 907300, angleproject:2712
Change-Id: I45a9f59babe90bbe2137d49c555884d7bbbcf170
Reviewed-on: https://chromium-review.googlesource.com/c/1382942
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
6 years ago
Bruce Dawson c6ffd7af7d Include ARM64 bits into win_toolchain/package_from_installed.py
These changes ensure that the packaged Windows SDK bits support targeting
ARM64 Windows. This change must not land until crrev.com/c/1339101 lands,
since it is needed to the varying number of vs_runtime_dll_dirs.

Bug: 893460
Change-Id: Ie32563067c6fb6078acfaccd6d3d572d1dd44888
Reviewed-on: https://chromium-review.googlesource.com/c/1330185
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
7 years ago
Henrique Ferreiro c5a26a769e [win-cross] Support using a zip file for the Windows SDK
Renames DEPOT_TOOLS_WIN_TOOLCHAIN_HTTP_BASE_URL to
DEPOT_TOOLS_WIN_TOOLCHAIN_BASE_LOCATION. This environment variable can
now point to a local directory where the Windows SDK zip file is stored.
This allows non-Googlers to cross-compile Chromium for Windows.

Bug: 852347
Change-Id: I00650e84247f35b4b8cfba204e0f2afd0882b69b
Reviewed-on: https://chromium-review.googlesource.com/1098256
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
7 years ago
Kimmo Kinnunen 54a81e45dd Add support for downloading Windows VS toolchain from a HTTP URL
Add support for downloading Windows VS toolchain from a HTTP URL.
This is useful for developers that do not have access to the
Chrome toolchain.

If the developer specifies DEPOT_TOOLS_WIN_TOOLCHAIN_HTTP_BASE_URL
environment variable, the toolchain will be downloaded from that
url.

Bug: 830569
Change-Id: I41d815a4460085c3b028f56f2e01adc68d8410fe
Reviewed-on: https://chromium-review.googlesource.com/1002892
Commit-Queue: Kimmo Kinnunen FI <kkinnunen@nvidia.com>
Reviewed-by: Scott Graham <scottmg@chromium.org>
7 years ago
Scott Graham 1e94e8933f win_toolchain: avoid dialog from GCM post git 2.14 on Windows
Git Credential Manager for Windows (as of git 2.14.1) really wants to
pop up a modal dialog. Setting this environment variable avoids this
so that the check for src-internal access can quietly fail.

Ref: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/482

Bug: 755694
Change-Id: I38aec008662fa0a6bccb0a6220d376063ee790e7
Reviewed-on: https://chromium-review.googlesource.com/617502
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
8 years ago
Dan Jacques 2a622f20ee [win_toolchain] Allow toolchain dir override.
Currently, the Windows toolchain is installed in a subdirectory under
"depot_tools". Allow this installation destination to be overridden,
either by flag (useful for testing) or via environment variable (to be
used in Chromium recipe).

BUG=chromium:727917
TEST=local
  - Ran explicitly w/ flag, seems to work.
  - Ran via Chromium's "vs_toolchain" with and without environment
    variable override, seems to install correctly.

Change-Id: I6b33832d7f099796e23da0548949073c70a17788
Reviewed-on: https://chromium-review.googlesource.com/521663
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
8 years ago
Bruce Dawson 93e1a76ec4 Actually enable crash dump collection on builders
Change crrev.com/1825163003 attempted to enable crash dump collection
on build machines, and it worked fine on local testing. However it only
worked because local testing was done using 64-bit Python. The builders
use python from depot_tools which is 32-bit Python so the changes all
went to "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft" instead of
to "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft", which means they were
ignored. For a year. This made investigation of linker crashes more
complicated than needed.

This change uses the necessary winreg.KEY_WOW64_64KEY dance so that the
64-bit registry is always used, whether running 32-bit Python or 64-bit
Python. Both versions were tested locally. The behavior on 32-bit
Windows is unknown but we don't support building on 32-bit Windows
anyway, and any failures would be rendered harmless by the try/except
block.

R=scottmg@chromium.org
BUG=704286

Change-Id: I6abc0e1e9c69b9a4e4b9c705bea9e4faadd0945c
Reviewed-on: https://chromium-review.googlesource.com/473567
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
8 years ago
Bruce Dawson b99bc3e190 Cleanup VS 2013/2015 confusion in toolchain script
The get-toolchain script still contained leftover goop for installing
the UCRT, even though that misguided plan was abandoned last year. This
change deletes that.

The get-toolchain script also confusingly used the vs2013_files
directory for anything that wasn't VS 2015. It doesn't technically make
any difference now that we use hashes for the toolchain directories, but
it was confusing when experimenting with VS 2017.

R=sebmarchand@chromium.org
BUG=683729

Change-Id: Ie0d3eccffe4796d4c5e23a28276acdd757e290d4
Reviewed-on: https://chromium-review.googlesource.com/453122
Reviewed-by: Sébastien Marchand <sebmarchand@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
8 years ago
scottmg 9fc458976c Don't try svn.chromium.org for src-internal check
svn://svn.chromium.org/chrome-internal/trunk/src-internal/ doesn't seem
to exist any more.

d:\src>"d:\src\depot_tools\svn_bin\svn.exe"  ls --non-interactive svn://svn.chromium.org/chrome-internal/trunk/src-internal/
svn: Can't connect to host 'svn.chromium.org': A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

R=brucedawson@chromium.org

Review-Url: https://codereview.chromium.org/2327213002
9 years ago
sebmarchand a0e66ea609 Limit the number of extra/missing files that get printed.
The message now look like this:

2 files missing from the 9ff97version of the toolchain:
        vs2013_files\9ff97\win_sdk\Include\10.0.10240.0\ucrt\assert.h
        vs2013_files\9ff97\win_sdk\Include\10.0.10240.0\ucrt\complex.h
22 extra files in the 9ff97version of the toolchain:
        vs2013_files\9ff97\win_sdk\Include\10.0.10240.0\ucrt\assert_.h
        vs2013_files\9ff97\win_sdk\Include\10.0.10240.0\ucrt\complex_.h
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\string\wmemmove_s.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\asctime.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\clock.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\ctime.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\days.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\difftime.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\ftime.cpp
        vs2013_files\9ff97\win_sdk\Source\10.0.10240.0\ucrt\time\gmtime.cpp
        ...

R=thakis@chromium.org

Review-Url: https://codereview.chromium.org/2092753003
9 years ago
sebmarchand e9013dbe6f Update the VS toolchain hash computation caching.
R=hans@chromium.org, scottmg@chromium.org
BUG=495204

Review-Url: https://codereview.chromium.org/2052533002
9 years ago
hans@chromium.org b380ad8370 Make "vs_toolchain.py update" work on case-sensitive file systems
Filenames must have their case preserved so the files can be opened,
but they need to be lowercased when hashed and sorted, to match the
expected hash.

In other words, this patch moves the lower-casing from GetFileList
(except to sort the list) to where filenames are fed into the hash.

BUG=495204

Review-Url: https://codereview.chromium.org/2039563002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@300696 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
sebmarchand@chromium.org 493ae64c3e Improve the toolchains-hash calculation.
I often ends up with a corrupt toolchain because I've used Windbg and there's now a bunch of PDB files in win_sdk/Debuggers/sym/... , I think that it's safe to ignore these files (the original package doesn't contain any of these files).

It's also useful to have more details about which files are missing and or which ones shouldn't be here when we get an invalid toolchain hash.

Finally, if you the hash of a toolchain doesn't correspond to the name of its root directory then it should be removed.

Review-Url: https://codereview.chromium.org/1974453003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@300557 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
brucedawson@chromium.org ff3af2c1e5 Enable crash dump collection on builders
Toolchain crashes on build machines are an ongoing problem, particularly
with the VC++ 2015 migration. Setting this registry key tells Windows
Error Reporting to record minidumps (a few MB typically) to a local
directory, up to a maximum of ten. This should help with investigations.

This change also suppresses Windows Error Reporting dialogs with the
DontShowUI registry value, to avoid builder hangs on crashes.

See also crrev.com/1816333002

BUG=440500

Review URL: https://codereview.chromium.org/1825163003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@299426 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
scottmg@chromium.org 0471355374 Make get_toolchain_if_necessary.py work on Mac again
R=thakis@chromium.org, sebmarchand@chromium.org
BUG=495204,585829

Review URL: https://codereview.chromium.org/1689723002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298720 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
brucedawson@chromium.org c4eb24a907 Skip installation of UCRT
The UCRT is now optional and it failed to install on one builder.
Making optional to fix build break

If this doesn't fix the break on the waterfall then the switch
to VS 2015 will need to be reverted (crrev.com/1598493004).

TBR=scottmg@chromium.org
BUG=440500

Review URL: https://codereview.chromium.org/1680433002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298634 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
scottmg@chromium.org 96ed03d55b Fix 'Calculating hash...' message
Right now I get

Calculating hash of toolchain in vs2013_files. Please wait...
Calculating hash of toolchain in vs2013_files. Please wait...
Calculating hash of toolchain in vs2013_files. Please wait...
Calculating hash of toolchain in vs2013_files. Please wait...
Calculating hash of toolchain in vs2013_files. Please wait...
Calculating hash of toolchain in vs2013_files. Please wait...

which isn't very informative (or at least a bit confusing as to why it's
doing the same thing 6 times).

R=brucedawson@chromium.org

Review URL: https://codereview.chromium.org/1671433002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298593 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago
sebmarchand@chromium.org a1798215d0 Add the possibility to keep several version of the VS toolchain.
BUG=

Review URL: https://codereview.chromium.org/1634923002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298557 0039d316-1c4b-4281-b951-d872f2087c98
9 years ago