Several tools, including the "git" recipe module, hard-code a
checkout-relative "git.bat" path. Git is a feature that is provided by
the system, both to tooling and recipes:
1) For users, "depot_tools" must be on PATH, and during setup it will
have installed "git.bat", ensuring that Git tooling is available in
PATH.
2) For bots, the system is responsible for providing "git.bat" on PATH.
This is typically done at "/b/depot_tools/git.bat", which is sync'd
through the "update_scripts" step.
By formally treating Git as a system resource, we absolve Windows bots
and users from manually installing a depot_tools-local Git, bringing
them in line with other platforms.
BUG=chromium:590806
TEST=local
Change-Id: I93e89855cdd330a2ba7a8cfb8117a1789d1ab54e
Reviewed-on: https://chromium-review.googlesource.com/568694
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This change should be safe, since batch files are not generally
write-contended, and "gclient" is run atomically (other things depend on
this).
The mechanism to detect whether we're downgrading is broken by
"bot_update". While future syncs should not encounter this issue,
current syncs still have no way to resolve it, and are left with a
broken "python.bat" setup.
We remedy this by always installing "python.bat", even if it's already
there. This is safe, since the intended state of HEAD is for this file
to be present at all times.
BUG=chromium:741650
TEST=None
Change-Id: Ief28217bf89d201d830c46b31e0b4040c51cb322
Reviewed-on: https://chromium-review.googlesource.com/568588
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
the revert mechanism for the Python bundle patch relies on the resitual
"python_bin_reldir.txt" file existing. Apparently "bot_update" purges
this file as part of its depot_tools checkout, breaking the revert
mechanism. Whitelist it so that this stops happening.
TBR=tandrii@chromium.org
BUG=chromium:741650
TEST=None
Change-Id: Id6b7c966b6fa3f5084be5e1906b3e10d38ab8b6c
Reviewed-on: https://chromium-review.googlesource.com/568530
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
This reverts commit bf1446791e.
Reason for revert: broke win bots http://o/410053
Original change's description:
> [win_tools] Use bundled Python CIPD packages.
>
> Enable bundled Python CIPD packages in bleeding-edge mode. This
> replaces the ZIP unpacking approach used before, and introduces
> validation and management through the CIPD tool. The bleeding edge
> version will only install if a sentinel file is present in the
> "depot_tools" root; otherwise, default behavior will continue.
>
> This method adds a upgrade and downgrade path to/from ZIP and
> CIPD installations. This is done by rewriting the "win_tools.bat"
> process:
>
> 1) Ensure that a bootstrap Python is present.
> 2) Use it to run "win_tools.py", which has the functionality of
> "git_bootstrap.py" plus Python installation.
> 3) Run "win_tools.py" with appropriate flags.
>
> Some tricks were employed to handle cases where there is an
> already-running Python instance that uses the current Python
> installation and executable. This happens on bots because the
> system uses the same "depot_tools" checkout at multiple launch
> layers. To this end, we use the "python.bat" as the "current Python"
> authority and refrain from cleaning up old Python directories if their
> "python.exe" binaries are currently in use.
>
> We change the Git bleeding edge file to share the same
> sentinel file as Python, ".bleeding_edge".
>
> The new Python should have the same facilities as the original Python
> bundle.
>
> BUG=chromium:740171
> TEST=local
>
> Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736
> Reviewed-on: https://chromium-review.googlesource.com/563036
> Commit-Queue: Daniel Jacques <dnj@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
TBR=iannucci@chromium.org,dnj@chromium.org
Change-Id: I84574a01bbad6596912e4aaa34f019d24720b638
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:740171
Reviewed-on: https://chromium-review.googlesource.com/567009
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
This makes bot_update query Gerrit when syncing for Gerrit tryjobs.
The query will establish the actual destination branch of the CL,
which can be different from master (e.g. a feature branch).
Bot_update will ensure to use this destination branch for the repo
that corresponds to the CL's project. Both the main project or a
deps'ed project work.
Initially, this lives behind a flag that can be controlled in
downstream recipes. Eventually we'll set this to default after a
gradual roll-out.
Branches in branch-heads are not supported yet.
Bug: 740456
Change-Id: I4a0d50e2ca8fe90f8d29964a3ffab17291f7be60
Reviewed-on: https://chromium-review.googlesource.com/566824
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Enable bundled Python CIPD packages in bleeding-edge mode. This
replaces the ZIP unpacking approach used before, and introduces
validation and management through the CIPD tool. The bleeding edge
version will only install if a sentinel file is present in the
"depot_tools" root; otherwise, default behavior will continue.
This method adds a upgrade and downgrade path to/from ZIP and
CIPD installations. This is done by rewriting the "win_tools.bat"
process:
1) Ensure that a bootstrap Python is present.
2) Use it to run "win_tools.py", which has the functionality of
"git_bootstrap.py" plus Python installation.
3) Run "win_tools.py" with appropriate flags.
Some tricks were employed to handle cases where there is an
already-running Python instance that uses the current Python
installation and executable. This happens on bots because the
system uses the same "depot_tools" checkout at multiple launch
layers. To this end, we use the "python.bat" as the "current Python"
authority and refrain from cleaning up old Python directories if their
"python.exe" binaries are currently in use.
We change the Git bleeding edge file to share the same
sentinel file as Python, ".bleeding_edge".
The new Python should have the same facilities as the original Python
bundle.
BUG=chromium:740171
TEST=local
Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736
Reviewed-on: https://chromium-review.googlesource.com/563036
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
We're implementing a bootstrap downgrade for potential revert of
https://chromium-review.googlesource.com/c/563036
Currently, the downgrade sees the presence of the landed patch
as a sign to completely reinstall Python. However, this causes
the "python276_bin" directory to be deleted, which both ruins
any running process (notably "service manager" and BuildBot)
and fails due to those running processes, leaving the system in
a broken state.
Instead, we'll view the revert path as a signal to swap in the
old "python.bat" specifically, then fall through to standard
installation detection. Since the upgrade will not actually
delete "python276_bin", the expected revert case will leave it
alone, preserving existing execution environments.
BUG=chromium:740966, chromium:740171
TEST=local
- Ran simulation of revert path with this change, observed
better outcome.
Change-Id: I0dfa5924a27bcaba49134272a344f7b9f1d475c5
Reviewed-on: https://chromium-review.googlesource.com/567167
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
The upcoming CL, https://chromium-review.googlesource.com/c/563036,
updates "bootstrap/win"'s mechanism. However, if that patch needs to be
reverted, its "python.bat" stub may still exist in the reverted
repository, and will continue to point to the reverted toolchain.
We don't have a good way to detect whether we should revert "python.bat".
Instead, we will look a file generated by the CL,
"//python_bin_reldir.txt". If we find it, we will assume that the updated
"python.bat" is in use, and reinstall.
Note that if that CL lands and does not require a revert, this logic
will be deleted. It's just there as a safety mechanism to allow a safe
revert.
BUG=chromium:740171
TEST=local
Change-Id: Ifc638cf0512d2a889c37fbf6b8e3f7a3269331b1
Reviewed-on: https://chromium-review.googlesource.com/566073
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This file is introduced via Puppet and will be migrated to from
".git_bleeding_edge". In the meantime, ignore it so Git doesn't
think it's an untracked modication.
TBR=iannucci@chromium.org
BUG=chromium:740171
TEST=None
Change-Id: Ia878996c3c32c49e1f7d80293bef55ed0a09dbab
Reviewed-on: https://chromium-review.googlesource.com/565888
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
The /a suffix in the hots URL is not necessary as the gerrit_util adds it by
default to authenticate. Having it hard-coded in the example can be misleading.
Bug: 685318
Change-Id: I333cd8b2aa9020aadfd186f2e18fbff0aa917681
Reviewed-on: https://chromium-review.googlesource.com/564611
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Support new fully-bundled Git CIPD packages. These are generated by the
"third_party_packages" infra builder, and require no installer.
BUG=chromium:740171
TEST=local
Change-Id: I6126655359ba981eb18ad1e088fe787100719d46
Reviewed-on: https://chromium-review.googlesource.com/562531
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
The -t tools in ninja fail if -j or -l are specified. So, autoninja.py
needs to watch for -t and omit -j and -l if it is noticed.
R=sebmarchand@chromium.org
BUG=740227
Change-Id: I1418193daeab154178d15be60ab09551bacaf3af
Reviewed-on: https://chromium-review.googlesource.com/563775
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Currently, the PowerShell execution does not allow STDIN to be
read by the CIPD client. Fix this by using PowerShell to acquire
the client, but batch to actually invoke the client. This removes
some indirection and allows STDIN to function as expected.
BUG=None
TEST=local
- Ran w/ and w/out ".cipd_client.exe" locally, observed
install and fallthrough.
- Ran w/ an error code, observed exit code being preserved.
- Ran "auth-login", was able to paste token.
Change-Id: I4efafa7dc80aa093de0fbf5dfd188c299fa104db
Reviewed-on: https://chromium-review.googlesource.com/563602
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Remove Windows SVN bootstrapping and some SVN tooling. Since
"depot_tools" is no longer sync'd to SVN, and we have been committed to
Git for years now, this is obsolete. Any transition code will never
reach SVN users, and any remaining code should not be used by Chromium
developers.
BUG=chromium:630904
TEST=unit
Change-Id: Ie984e8400a748702b125eaeed8157719ef4b88cc
Reviewed-on: https://chromium-review.googlesource.com/562748
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Due to relaxation of when last paragraph of commit message is
is consider as containing footers, `git cl land` started removing
non-canonic footer lines from last paragraph if Git Numberer is enabled
on a repo. This only manifests in manual lands of Rietveld CLs or
bypassing code review entirely.
R=agable@chromium.org
Bug: 736852
Change-Id: I3972c590c3959974157ada9de9891a3c08bd385a
Reviewed-on: https://chromium-review.googlesource.com/562278
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
It's currently possible for CIPD bootstraps that provision concurrently to:
1) On Linux, step on each other during download, and
2) On Windows, fail.
Fix these respective scripts so that bootstraps are safe to use
concurrently.
On Linux and Mac, we download to a temporary file and use "mv" (atomic)
to write it to the final destination. Concurrent initializations will
perform parallel downloads, execute the "mv", and copy their downloaded
file to the destination path.
On Windows, we use filesystem locking to lock the operation and ensure
that only one download can happen.
BUG=chromium:739195
TEST=local
- Ran in parallel on Windows, Linux, and Max.
Change-Id: Ie050d37598da67389f21728e781bd58904ef9c17
Reviewed-on: https://chromium-review.googlesource.com/560521
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Instead of passing around 6+ out-parameters, keep state inside the class.
No intended behavior change.
Bug: 570091
Change-Id: Ia15c1db2170680f0ec087b61f51af49e12fd7579
Reviewed-on: https://chromium-review.googlesource.com/561700
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
It turns out it most likely won't be needed. Instead, auto-pinning
deps is useful mode.
This removal helps simplify and restructure the flatten code.
Bug: 570091
Change-Id: Ia70221c88a37f210cf1338a36d79168d2cfe96f6
Reviewed-on: https://chromium-review.googlesource.com/561717
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
This'll allow to query gerrit from recipes for e.g. current changes in
CQ.
Bug: 685318
Change-Id: I73d08d4b186b2e5fe044fd4d4fafd9db62e27066
Reviewed-on: https://chromium-review.googlesource.com/558939
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
The main goal is to provide a migration path, where we can add
conditional src-internal entry to src/DEPS, and have it still
work on checkouts where .gclient pulls src-internal, provided
the condition evaluates to False.
The migration path is then to remove the .gclient entry, and
at the same time flip the condition to True (e.g. by overriding
a variable).
Bug: 570091
Change-Id: I9b9850a644463ab0b1f368d65a5cd5f47cf7be97
Reviewed-on: https://chromium-review.googlesource.com/559150
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Make Linux/Mac CIPD wrapper fail if any step returns an error.
Include the "-f" curl flag, which will cause it to error on non-200
status.
BUG=chromium:739195
TEST=local
R=iannucci@chromium.org
Change-Id: I3efea3f38c1ee322cb9d63f29819472564107ddc
Reviewed-on: https://chromium-review.googlesource.com/560723
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
The previous code did not distinguish between None and [] value for orig_deps.
This could be triggered by not having any deps, but some entries in deps_os,
in which case these deps_os entries would get added by mistake to deps.
Bug: 570091
Change-Id: Id5b41e37ea717d5d00404694ad5bfce8df9da934
Reviewed-on: https://chromium-review.googlesource.com/558349
Reviewed-by: Michael Moss <mmoss@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
"bot_update" currently passes a full "gclient" specification as a
command-line argument to "bot_update.py". This file contains all sorts
of junk that clutters the command-line, and revealed a batch file
escaping bug that seems to be present in the recipe engine. Change this
so that "bot_update" passes the spec as file content instead of a
command-line flag.
Also, "bot_update" apparently called "--spec", which optparse kindly
recognized as the real flag, "--specs". This has always(?) been a bug,
and is no longer a problem.
BUG=chromium:738348
TEST=expectations
Change-Id: I5324b94dd5ef78503e7cfdc28b5fde8f434e4359
Reviewed-on: https://chromium-review.googlesource.com/558207
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Prefix "depot_tools" to PATH instead of appending. This will ensure that
"depot_tools" pulls resources from the recipe checkout as a preference,
not as a last resort.
Use "env_prefixes" instead of "env" to prefix PATH.
Remove Git speed limit settings, which are taken care of by the Git
wrapper now.
BUG=chromium:738176
TEST=expectations
R=hinoka@chromium.org, iannucci@chromium.org
Change-Id: I2e509a331266e867be323bc70e3bc0f08f85246f
Reviewed-on: https://chromium-review.googlesource.com/556287
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Dan Jacques
Dan Jacques
Andrii Shyshkalov
Michael Achenbach
recipe-roller
Paweł Hajdan, Jr
Nicolas Dossou-gbete
Bruce Dawson
Anna Henningsen
Vadim Shtayura
Robert Iannucci