It now fetches and installs packages in parallel. This has a side
effect that the order of unzipping of packages in a manifest is
non-deterministic and depends on when particular packages finish
downloading.
To be able to show progress of parallel operations there's also
a new terminal UI with progress bars. It activates when running
CIPD at `info` logging level (default) if the stderr is attached
to a terminal (i.e. not piping to a file or the recipe engine).
Two new env vars control the new behavior:
* CIPD_PARALLEL_DOWNLOADS: set to 0 to completely disable
parallelism and restore pre 2.6.0 behavior. Set to 1 to fetch
and unzip packages in paralle, but fetch at most one package
at once. Default is 4.
* CIPD_SIMPLE_TERMINAL_UI: set to 1 to use log-based UI instead
of progress bars.
R=iannucci@chromium.org
BUG=1217435
Change-Id: I16815dbc397137fbb8ef09e09d0a04c77660c0c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2993750
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
All packages except mac-amd64 ones are built with go1.16. mac-amd64
are built with go1.15, to preserve OSX 10.10/10.11 compatibility.
Note that we still use mac-amd64 even on arm64 OSX, since
the bootstrap script still installs mac-amd64 version of CIPD.
R=iannucci@chromium.org, tandrii@chromium.org
BUG=1102967
Change-Id: I8fdb27397fbbf238fe39e81305d4642a0500ab33
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2699690
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
It was built with go1.15 which has a bug in deflate implementation
which may result in corruption when uploading >2GB files.
R=iannucci@chromium.org
BUG=1129704
Change-Id: Id61eaaed5dd18d8e51ac5150a9d29a5825805a17
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2418899
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Includes bug fix for crbug.com/936911 and no other changes.
R=tandrii@chromium.org, iannucci@chromium.org
BUG=936911
Change-Id: I92e6725dc16b513579db87b81e389f9d61bdf62b
Reviewed-on: https://chromium-review.googlesource.com/c/1496034
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Includes a bug fix for incorrect handling of subcommands with typos in them. No
other changes.
BUG=931755
R=iannucci@chromium.org
Change-Id: I4a3c523452939bdb5a6f26637f1f7b4a2eeb4b0c
Reviewed-on: https://chromium-review.googlesource.com/c/1474373
Auto-Submit: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
The most notable change is in how the integrity of the tag cache file is
checked. Once deployed, existing tag cache files will be considered invalid and
wiped.
The only observable side effect is the following line in the log:
"can't deserialize tag cache - no sha256 is recorded in the file"
R=tandrii@chromium.org, iannucci@chromium.org
Change-Id: I7ea300a4e7ad8be0c1d42ae561c5202420d2db62
Reviewed-on: https://chromium-review.googlesource.com/c/1327823
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
The new version uses SHA256 for uploads by default. It also fully supports
$ResolvedVersions and 'selfupdate-roll'.
The roll was generated by:
$ ./cipd selfupdate-roll \
-version-file cipd_client_version \
-version git_revision:fb963f0f43e265a65fb7f1f202e17ea23e947063
BUG=821194, 870166
R=nodir@chromium.org, iannucci@chromium.org
Change-Id: If7c23c43ff2c1bf50c9c50f4e9e17a23e84859b9
Reviewed-on: https://chromium-review.googlesource.com/1227431
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Linux and OSX only for now. This also rolls CIPD client to a version that
supports pinned hashes (v2.2.5).
CIPD_CLIENT_VER and CIPD_CLIENT_SRV are no longer supported as env vars, since
it makes no sense when pinning hashes of the binaries at specific version on
the specific backend.
Also somewhat cleanup 'cipd' script to use "${VAR}", stderr and colored output
consistently.
R=iannucci@chromium.org, nodir@chromium.org
BUG=870166, 874586
Change-Id: Iac67fbb6b5d07dcd81d44536737b03b146f1ad14
Reviewed-on: https://chromium-review.googlesource.com/1176727
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
This reverts commit eebc3d8232.
Reason for revert: crbug.com/874586
Original change's description:
> [cipd] Check CIPD client hash against pinned SHA256 during updates.
>
> Linux and OSX only for now. This also rolls CIPD client to a version that
> supports pinned hashes (v2.2.5).
>
> CIPD_CLIENT_VER and CIPD_CLIENT_SRV are no longer supported as env vars, since
> it makes no sense when pinning hashes of the binaries at specific version on
> the specific backend.
>
> Also somewhat cleanup 'cipd' script to use "${VAR}", stderr and colored output
> consistently.
>
> R=iannucci@chromium.org, nodir@chromium.org
> BUG=870166
>
> Change-Id: I9e61f9f8fbdcf10985c52828b2bfbec64b4234f0
> Reviewed-on: https://chromium-review.googlesource.com/1171957
> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
TBR=iannucci@chromium.org,vadimsh@chromium.org,nodir@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: 870166
Change-Id: I9aa8e7a7f07520aa69d366c76e4dbccae345bc00
Reviewed-on: https://chromium-review.googlesource.com/1175294
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Linux and OSX only for now. This also rolls CIPD client to a version that
supports pinned hashes (v2.2.5).
CIPD_CLIENT_VER and CIPD_CLIENT_SRV are no longer supported as env vars, since
it makes no sense when pinning hashes of the binaries at specific version on
the specific backend.
Also somewhat cleanup 'cipd' script to use "${VAR}", stderr and colored output
consistently.
R=iannucci@chromium.org, nodir@chromium.org
BUG=870166
Change-Id: I9e61f9f8fbdcf10985c52828b2bfbec64b4234f0
Reviewed-on: https://chromium-review.googlesource.com/1171957
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
New packages are built using SHA256 as instance ID (instead of SHA1).
R=iannucci@chromium.org
BUG=821194, 867819, 813782
Change-Id: I61cf71386975725f7f63097eeb62f094ff50e396
Reviewed-on: https://chromium-review.googlesource.com/1165598
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
2018-06-26 vadimsh [cipd] Do not barf on -pkg-var when using -in.
2018-06-26 vadimsh [cipd] Fix and improve output of ensure-file-verify.
2018-06-26 vadimsh [cipd] Fix hanging in 'ensure-file-verify' if the ensure file is broken.
R=iannucci@chromium.org, tandrii@chromium.org
BUG=801423
Change-Id: Ic48753e30f2412f214ebfb73bb7693e5834cdbf7
Reviewed-on: https://chromium-review.googlesource.com/1115464
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
This allows the cipd client to process refs with slashes in them.
Also it is compiled with cgo on OSX, which should help with DNS
resolution errors that happen on some machines.
R=vadimsh@chromium.org
Bug: 811974, 803673
Change-Id: If298db5cfe0e10e3734deb4215bf82b95ff5e09e
Reviewed-on: https://chromium-review.googlesource.com/940521
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
No code changes, just rebuilt with 1.9.4 that includes potential fix for
deadlocks on Windows.
R=iannucci@chromium.org
BUG=810072
Change-Id: Ic39de760657e86abb5667b6fd9d200e74b140aeb
Reviewed-on: https://chromium-review.googlesource.com/909725
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Notable changes here:
* 'cipd' binary is now statically linked and doesn't depend of libc.so.
* New 'cipd instances <pkg>' subcommand that list recently uploaded instances
of a package.
* cipd now properly removes empty directory trees if they are left after
uninstalling some files.
R=iannucci@chromium.org
BUG=796320, 789750
Change-Id: Iebc5655640defbbef4a5fb1850aeb852746e5ef3
Reviewed-on: https://chromium-review.googlesource.com/835657
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This is best-effort only; s390x is still not a supported platform.
CIPD pin bump contains no client changes; just points to a version
which is built for s390x as well our normal platforms.
R=dnj@chromium.org, vadimsh@chromium.org
Bug: 764087
Change-Id: I777246712eea1b39036f3b14b88cbd41c7cf0507
Reviewed-on: https://chromium-review.googlesource.com/661787
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
e130567e [cipd] Add spacers to help text to make it more readable.
91b1bda5 [cipd] Enable package name templates for all CLI functions
a10ef48a [cipd] Move template expansion code to common.
4c613c75 cipd: Support template expansion in @subdir
3847616b Rename github.com/luci/{luci-go,gae} to go.chromium.org/{luci,gae}.
R=vadimsh@chromium.org
Bug: 756288
Change-Id: Ie40daf2d80ea715fb303eb199b6c5abf8b4870ef
Reviewed-on: https://chromium-review.googlesource.com/619966
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Bump the CIPD client version to
b482bfa19d3fa79b4b40dd5ac7a34479052e3df1. This version inculdes support
for the "-log-level" flag.
Update CIPD bootstrap scripts and tool installation scripts to log at
warning level, so that standard operation doesn't produce any logs.
This version differs from the previous CL in that the "selfupdate"
command no longer uses "-log-level". This fixes an issue where the
current CIPD, which doesn't support "-log-level", is given the flag
prior to actually updating to a version that does.
BUG=chromium:748651
TEST=canary
- Tested this CIPD version on canary waterfall successfully.
Change-Id: I69dfe522d890eb307214f18d0467ce921a766446
Reviewed-on: https://chromium-review.googlesource.com/589568
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
This reverts commit 18aa2d93ed.
Reason for revert: <INSERT REASONING HERE>
Original change's description:
> [cipd] Bump version, log at warning.
>
> Bump the CIPD client version to
> e2a98b41adb3403b5b7771c4101b42a55dc2dd9e. This version inculdes support
> for the "-log-level" flag.
>
> Update CIPD bootstrap scripts and tool installation scripts to log at
> warning level, so that standard operation doesn't produce any logs.
>
> BUG=chromium:748651
> TEST=canary
> - Tested this CIPD version on canary waterfall successfully.
> - Tested "gclient" bootstrap and "vpython" / "led" on Windows system,
> no output and works.
> - Tested "vpython" / "led" on OSX system, no output and works.
>
> Change-Id: Iae316908283c43c504f293aafdf897df83dccf8d
> Reviewed-on: https://chromium-review.googlesource.com/588889
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Commit-Queue: Daniel Jacques <dnj@chromium.org>
TBR=iannucci@chromium.org,dnj@chromium.org
Change-Id: Ia611c1aeb204c609ff3469076b8256c54e702230
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:748651
Reviewed-on: https://chromium-review.googlesource.com/589527
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Bump the CIPD client version to
e2a98b41adb3403b5b7771c4101b42a55dc2dd9e. This version inculdes support
for the "-log-level" flag.
Update CIPD bootstrap scripts and tool installation scripts to log at
warning level, so that standard operation doesn't produce any logs.
BUG=chromium:748651
TEST=canary
- Tested this CIPD version on canary waterfall successfully.
- Tested "gclient" bootstrap and "vpython" / "led" on Windows system,
no output and works.
- Tested "vpython" / "led" on OSX system, no output and works.
Change-Id: Iae316908283c43c504f293aafdf897df83dccf8d
Reviewed-on: https://chromium-review.googlesource.com/588889
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>
New version skips unnecessary hash check for files that are already
in the cache.
R=iannucci@chromium.org
BUG=
Change-Id: Icdbd1d171fa77cf7e353c84e82ded8eafd88c88d
Reviewed-on: https://chromium-review.googlesource.com/454241
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Vadim Shtayura
Matt Kotsenas
Robert Iannucci
Andrii Shyshkalov
Sergey Berezin
Dan Jacques
Dan Jacques