gclient-sync + git-cache are racy because if two different DEPS files
dependent on the same repo URL, even if depsed to different checkout
dirs, they use same git cache dir. If both checkout dirs are synced at
the same time, one of them cannot acquire a lock.
This is a cheap solution to change --lock_timeout option default value
from 0 to 5m.
R=hinoka@chromium.org
BUG=593468
Review URL: https://codereview.chromium.org/1785083005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@299386 0039d316-1c4b-4281-b951-d872f2087c98
I guess I'm the only developer using git-cache, which is sad.
Hopefully these fixes will make it easier to adapt this to developer
usage some time in the FUTURE.
BUG=583420
TEST=Works for me
R=agable@chromium.org,tandrii@chromium.org
Review URL: https://codereview.chromium.org/1650993005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@298531 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
breaks git-bash on windows
Original issue's description:
> git_cl/gclient: use python2
>
> Newer distros are defaulting /usr/bin/python to python3, so use python2
> explicitly so we continue working.
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=297535
TBR=stip@chromium.org,sergeyberezin@chromium.org,vapier@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/1442583004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@297544 0039d316-1c4b-4281-b951-d872f2087c98
Newer distros are defaulting /usr/bin/python to python3, so use python2
explicitly so we continue working.
Review URL: https://codereview.chromium.org/1437773002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@297535 0039d316-1c4b-4281-b951-d872f2087c98
The newer form works with python2 and python3.
Review URL: https://codereview.chromium.org/1436763002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@297517 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code work both in python2 and python3.
Review URL: https://codereview.chromium.org/1433153003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@297516 0039d316-1c4b-4281-b951-d872f2087c98
This helps prevent behavior such as setting '--name .' so that the .gclient
file contains '.' as the name of the solution, and the .gclient file ends
up a sibling of the .git file, rather than a sibling of the git checkout's
containing directory.
R=iannucci@chromium.org, thestig@chromium.org
Review URL: https://codereview.chromium.org/1406053003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@297343 0039d316-1c4b-4281-b951-d872f2087c98
This cl ensures that gclient enforces sane revision overrides and
onfiguration when running a revert, just like it does on a normal sync/update.
R=iannucci@chromium.org
BUG=420919
Review URL: https://codereview.chromium.org/1160113003.
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@295571 0039d316-1c4b-4281-b951-d872f2087c98
Handle KeyboardInterrupt gracefully rather the printing a
backtrace. Most users of these tools don't expect a
backtrace when then hit Ctrl-C.
Also, fix a few other inconsistencies found in the python
startup code of these different scripts:
- always call main function 'main' (rather than 'Main')
- always return 0 from main function
- if main takes args never include argv[0]
Review URL: https://codereview.chromium.org/955993006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294250 0039d316-1c4b-4281-b951-d872f2087c98
This adds the '--auto_rebase' sync option, enabling parallel updates to
automatically rebase local Git branches during sync.
BUG=None
TEST=local
Review URL: https://codereview.chromium.org/930693002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294151 0039d316-1c4b-4281-b951-d872f2087c98
Reland crrev.com/743083002, which was reverted in crrev.com/796053002
due to some test flakiness, probably related with an old version of Git on
the bots. Relanding now that the infra has been updated to Trusty (plus
adding some de-flake precautions).
Original CL Description:
Make gclient ready for the Blink (DEPS to main project) transition
This CL makes gclient understand correctly whether a git project is
being moved from DEPS to an upper project and vice-versa.
The driving use case for this is the upcoming Blink merge, where
third_party/Webkit will be removed from DEPS (and .gitignore) and will
become part of the main project.
At present state, gclient leaves the .git folder around when a project
is removed from DEPS, and that causes many problems.
Furthermore this CL solves the performance problem of bisecting across
the merge point. The subproject's (Blink) .git/ folder is moved to a
backup location (in the main checkout root) and is restored when moving
backwards, avoiding a re-fetch when bisecting across the merge point.
BUG=431469
Review URL: https://codereview.chromium.org/910913003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@294082 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
The test intoruced is flake (see crrev.com/753543006/).
Need to look into that with more patience as it reproduces only on precise.
Original issue's description:
> Make gclient ready for the Blink (DEPS to main project) transition
>
> This CL makes gclient understand correctly whether a git project is
> being moved from DEPS to an upper project and vice-versa.
> The driving use case for this is the upcoming Blink merge, where
> third_party/Webkit will be removed from DEPS (and .gitignore) and will
> become part of the main project.
>
> At present state, gclient leaves the .git folder around when a project
> is removed from DEPS, and that causes many problems.
>
> Furthermore this CL solves the performance problem of bisecting across
> the merge point. The subproject's (Blink) .git/ folder is moved to a
> backup location (in the main checkout root) and is restored when moving
> backwards, avoiding a re-fetch when bisecting across the merge point.
>
> BUG=431469
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=293329
TBR=iannucci@chromium.org,maruel@chromium.org,jochen@chromium.org
NOTREECHECKS=true
NOTRY=true
BUG=431469
Review URL: https://codereview.chromium.org/796053002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293352 0039d316-1c4b-4281-b951-d872f2087c98
This CL makes gclient understand correctly whether a git project is
being moved from DEPS to an upper project and vice-versa.
The driving use case for this is the upcoming Blink merge, where
third_party/Webkit will be removed from DEPS (and .gitignore) and will
become part of the main project.
At present state, gclient leaves the .git folder around when a project
is removed from DEPS, and that causes many problems.
Furthermore this CL solves the performance problem of bisecting across
the merge point. The subproject's (Blink) .git/ folder is moved to a
backup location (in the main checkout root) and is restored when moving
backwards, avoiding a re-fetch when bisecting across the merge point.
BUG=431469
Review URL: https://codereview.chromium.org/743083002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@293329 0039d316-1c4b-4281-b951-d872f2087c98
This is breaking the ability to specify an alternate DEPS file, and do
things like sync release tags where DEPS is still from svn and we need
.DEPS.git to do a git checkout.
R=iannucci@chromium.org, szager@chromium.org
BUG=407049
Review URL: https://codereview.chromium.org/511533003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@291665 0039d316-1c4b-4281-b951-d872f2087c98
This is primarily a hack to enable the chromeos ebuild.
BUG=367027
TBR=iannucci@chromium.org,mmoss@chromium.org
Review URL: https://codereview.chromium.org/490233003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@291075 0039d316-1c4b-4281-b951-d872f2087c98
Original CL: https://codereview.chromium.org/440263002/
Many people* have complained on chromium-dev about the long times
required to perform a full fetch over a DSL. This seems to be mostly
due to the huge size of chromium's history (~9 GB). On the other side,
not everybody is interested in downloading the full git history of
the projects. The size of git packs required to fetch a working HEAD
is one order of magnitude smaller (1.5 GB).
This change makes it possible to perform a shallow fetch (in a way
which is consistent with DEPS, leveraging git templates on clone),
reducing fetch times by 80% for those not interested in the history.
* See:
[chromium-dev] "fetch chromium" keeps hanging/getting stuck on Windows 7
[chromium-dev] Initial checkout with git taking long
[chromium-dev] Trying to get latest source code fails when fetching
[chromium-dev] Gclient sync takes too long
BUG=228996
TBR=iannucci@chromium.org,szager@chromium.org,wtc@chromium.org
Review URL: https://codereview.chromium.org/440273002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@287793 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
Broke the WebRTC waterfall:
http://build.chromium.org/p/tryserver.webrtc/builders/win/builds/3958/steps/gclient%20revert/logs/stdio
Original issue's description:
> Add --no-history option to fetch and gclient for shallow clones.
>
> Many people* have complained on chromium-dev about the long times
> required to perform a full fetch over a DSL. This seems to be mostly
> due to the huge size of chromium's history (~9 GB). On the other side,
> not everybody is interested in downloading the full git history of
> the projects. The size of git packs required to fetch a working HEAD
> is one order of magnitude smaller (1.5 GB).
> This change makes it possible to perform a shallow fetch (in a way
> which is consistent with DEPS, leveraging git templates on clone),
> reducing fetch times by 80% for those not interested in the history.
>
> * See:
> [chromium-dev] "fetch chromium" keeps hanging/getting stuck on Windows 7
> [chromium-dev] Initial checkout with git taking long
> [chromium-dev] Trying to get latest source code fails when fetching
> [chromium-dev] Gclient sync takes too long
>
> BUG=228996
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=287606
TBR=iannucci@chromium.org,szager@chromium.org,wtc@chromium.org
NOTREECHECKS=true
NOTRY=true
BUG=228996
Review URL: https://codereview.chromium.org/440263002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@287637 0039d316-1c4b-4281-b951-d872f2087c98
Many people* have complained on chromium-dev about the long times
required to perform a full fetch over a DSL. This seems to be mostly
due to the huge size of chromium's history (~9 GB). On the other side,
not everybody is interested in downloading the full git history of
the projects. The size of git packs required to fetch a working HEAD
is one order of magnitude smaller (1.5 GB).
This change makes it possible to perform a shallow fetch (in a way
which is consistent with DEPS, leveraging git templates on clone),
reducing fetch times by 80% for those not interested in the history.
* See:
[chromium-dev] "fetch chromium" keeps hanging/getting stuck on Windows 7
[chromium-dev] Initial checkout with git taking long
[chromium-dev] Trying to get latest source code fails when fetching
[chromium-dev] Gclient sync takes too long
BUG=228996
Review URL: https://codereview.chromium.org/437903002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@287606 0039d316-1c4b-4281-b951-d872f2087c98
Also cast recursedeps to a set. We'll continue
to support using a list literal form in DEPS for
now.
R=iannucci@chromium.org,agable@chromium.org
BUG=393000
Review URL: https://codereview.chromium.org/385123008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@283474 0039d316-1c4b-4281-b951-d872f2087c98
Created the 'GitRefish' class to centralize 'git' refish parsing and consistent
usage by 'gclient' 'git' code.
BUG=373504
TEST=localtest
R=agable@chromium.org, iannucci@chromium.org
Review URL: https://codereview.chromium.org/328843005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@281553 0039d316-1c4b-4281-b951-d872f2087c98
Now that recurselist is no longer a list, it doesn't make
sense to call it recurselist. recurseset is available, but
that's not as easy to read/say compared to recurselist.
Call this recursedeps, instead.
R=iannucci@chromium.org
BUG=390246
Review URL: https://codereview.chromium.org/363103002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@281107 0039d316-1c4b-4281-b951-d872f2087c98
It's possible to tell gclient to use a different
"deps" file from the default DEPS through the "deps_file"
variable in the .gclient file.
If this file is missing, fallback to DEPS (the
default).
BUG=390700
Review URL: https://codereview.chromium.org/368713002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@280921 0039d316-1c4b-4281-b951-d872f2087c98
Previously, recursion overrides were only available by setting
a numeric 'depth' value in a DEPS file. This meant that it
was not possible to control recursion per-dependency entry.
This change adds a recurselist variable with a list structure.
If a named dependency is present in the list, then gclient will
recurse into that dependency's DEPS.
As part of this change, I move the recursion controls off of
DependencySetting and onto Dependency. The new setup of being
based on Dependency allows access to the dependency's name.
The controls are only called from Dependency instances. They
have always needed access to self.parent (in the Dependency
context), so this should be more correct than the previous setup.
BUG=390246
Review URL: https://codereview.chromium.org/331373009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@280690 0039d316-1c4b-4281-b951-d872f2087c98
Currently, only top-level solutions may be pinned via --revision. With this, we can:
gclient sync --revision src/third_party/WebKit@deadbeef
BUG=
Review URL: https://codereview.chromium.org/290443003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@271571 0039d316-1c4b-4281-b951-d872f2087c98
This occurs for svn checkouts when the DEPS change from:
third_party/skia/gyp
third_party/skia/include
third_party/skia/src
to:
third_party/skia
First, gclient deletes/moves third_party/skia to make room for the new checkout.
When finished, it notices that the three subdirectories under skia are now
"unversioned" according to .gclient_entries, even though they are part of Skia
repo. On the bots, this causes them to be deleted.
There is a check to determine whether the thought-to-be unversioned directories
are actually subdirectories of another checkout, but it doesn't work in the svn
case because the check assumes that, if they are subdirectories of another
checkout, it must be an svn checkout. scm.GetCheckoutRoot() returns None
because it can't find an svn checkout containing those subdirectories. None is
not in gclient_entries, so the directories get deleted.
BUG=
Review URL: https://codereview.chromium.org/275103009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@270133 0039d316-1c4b-4281-b951-d872f2087c98
So that bot_update can use them to figure out which repo correspons to where.
BUG=370503
Review URL: https://codereview.chromium.org/268373003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@268875 0039d316-1c4b-4281-b951-d872f2087c98
Evidence indicates that running non-builtin git commands is very
slow in msysgit, slow enough to dominate the running time of
gclient sync. With this change, gclient never shells out to
git-cache; it import the lib directly instead.
R=agable@chromium.org,hinoka@chromium.org
BUG=
Review URL: https://codereview.chromium.org/229653002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@262759 0039d316-1c4b-4281-b951-d872f2087c98
Features:
- Non-verbose output is now limited to a one-line progress
indicator.
- Verbose output is now collated per subprocess. As soon as a
subprocess finishes, its full output is dumped to terminal.
- Verbose output is prefixed with timestamps representing elapsed
time since the beginning of the gclient invocation.
- git progress indicators ("Receiving objects", etc.) are limited to
one line every 10 seconds.
- In both verbose and non-verbose mode, if a failure occurs, the
full output of the failed update operation is dumped to terminal
just before exit.
- In the event that updates are progressing, but slowly,
"Still working" messages will be printed periodically, to pacify
users and buildbots.
BUG=
R=hinoka@google.com
Review URL: https://codereview.chromium.org/227163002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@262500 0039d316-1c4b-4281-b951-d872f2087c98
Instead of having custom logic for dealing with cache directories, use
git_cache.py to populate caches.
Also fixes a bug in git_cache.py where it was looking for lockfiles in cwd rather than the cache dir.
Other changes:
* _Run now returns output.
* Always print to stdout in CheckCallAndFilterOutput, even if it gets a carriage return. This is done because git progress report are carriage returns and not newlines and we don't want everything on the same line and not strip out the CRs.
* Removed members changed tests, its not very useful to know a new import is added.
BUG=339171
Review URL: https://codereview.chromium.org/180243006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@254248 0039d316-1c4b-4281-b951-d872f2087c98
It's not true (anymore?) that the number of SCM jobs for gclient sync equals the
number of cores, thus the new text is less specific.
BUG=none
Review URL: https://codereview.chromium.org/184733007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@254242 0039d316-1c4b-4281-b951-d872f2087c98
Given that we search for the gclient root in ancestor directories based on the file name, specifying a file with separators doesn't make a lot of sense.
BUG=336946
R=iannucci@chromium.org
Review URL: https://codereview.chromium.org/150633004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@249077 0039d316-1c4b-4281-b951-d872f2087c98
Reason for revert:
This breaks `gclient sync` for me. Before this patch:
________ found .git directory; skipping src
After this patch:
Error: 1> Can't update/checkout /Volumes/MacintoshHD2/src/chrome-git/src if an unversioned directory is present. Delete the directory and try again.
Original issue's description:
> If the destination directory doesn't contain the desired repo, delete it
>
> Basic functionality - this may not be in quite the right place.
>
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=245404
TBR=iannucci@chromium.org,ilevy@chromium.org,cmp@chromium.org,maruel@chromium.org,morrita@chromium.org,borenet@google.com
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/141633005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@245530 0039d316-1c4b-4281-b951-d872f2087c98
Basic functionality - this may not be in quite the right place.
Review URL: https://codereview.chromium.org/61623008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@245404 0039d316-1c4b-4281-b951-d872f2087c98
It's possible in gclient to list multiple operating systems on the
command line and while handling them all perfectly might not be possible
it's possible to make it better than today.
This patch makes sure None never overrides any previous value and it
adds some output to indicate what kind of dangerous choices it made.
BUG=248168
Review URL: https://codereview.chromium.org/23875029
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@240892 0039d316-1c4b-4281-b951-d872f2087c98
These are run for a given dependency after it has been synced but before its
DEPS have been synced. This will help to switch Chromium to depend on Skia's
git repository (skia:1638).
Review URL: https://codereview.chromium.org/25322002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@228651 0039d316-1c4b-4281-b951-d872f2087c98
This is in order to support 'features' like got_revision for build systems
unwilling to scrape the human-readable output of gclient in order to extract
basic information :)
R=agable@chromium.org, maruel@chromium.org
BUG=265618
Review URL: https://chromiumcodereview.appspot.com/23753008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@225046 0039d316-1c4b-4281-b951-d872f2087c98
Update subcommand to support examples, always disable format_description and
format_epilog and add colors when enabled.
R=iannucci@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/22824018
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@218180 0039d316-1c4b-4281-b951-d872f2087c98
Convert monkey patching to class inheritance for OptionParser usage in:
commit_queue.py, gclient.py and trychange.py.
Monkey patching confuses the hell out of pylint.
R=jochen@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/19552004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@213423 0039d316-1c4b-4281-b951-d872f2087c98
We can use this to evaluate the usefulness of making hooks run in parallel.
Review URL: https://chromiumcodereview.appspot.com/18851005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@211446 0039d316-1c4b-4281-b951-d872f2087c98
Sometimes we wish to pull in a complicated dependency but want to
suppress or replace one or more of the hooks rules. Say for example
we want to use a different way of generating the projects, or
have a different set of landmine expectations.
Here we add a custom_hooks section mirroring custom_deps to allow us to
override sections we have identified in the DEPS file. To do so,
we add an optional name to the elements of the hooks list, and overwrite
those whose name matches.
Conventions between included DEPS and the .gclient as to the meanings of the
name are equivalent to the meaning of the customized deps, and so do not
benefit from further structure or definition.
BUG=None
TEST=local unit test
Review URL: https://chromiumcodereview.appspot.com/17742004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@210573 0039d316-1c4b-4281-b951-d872f2087c98
This enables significant time savings, especially since file_list only exists to
enable file-specific hooks (which, AFAIK, nothing actually uses). On a z620
(linux) using the cached git repos, a first-time `gclient sync --nohooks` takes:
* (with) 131.06s user 14.10s system 117% cpu 2:03.89 total
* (without) 482.13s user 189.35s system 144% cpu 7:45.63 total
This change makes nohooks cause file_list to be None if we don't need to
accumulate it, and updates GitWrapper and SvnWrapper appropriately.
R=szager@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/18541006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@210026 0039d316-1c4b-4281-b951-d872f2087c98
Instead of cloning straight into place, clones are made to a global cache dir,
and then local (using --shared) clones are made from the cache to the final
resting place. This means the 'final' clones are full repos with no shenanigans,
meaning that branches, commits, etc. all work, which should allow the rest of
the gclient ecosystem to work without change as well.
The primary benefit is, of course, reduced network IO, and a much lower cost for
'clobber' operations (assuming we don't clobber the cache). It also means that
a given bot can have a greater number of checkouts, since the entire git history
will only be stored once per machine, instead of once per checkout.
R=dpranke@chromium.org, szager@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/18328003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@210024 0039d316-1c4b-4281-b951-d872f2087c98
This will be used on the bots when calling apply_patch, which commits to git
repos.
R=dpranke@chromium.org,szager@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/18006002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@209040 0039d316-1c4b-4281-b951-d872f2087c98
This will speed up main waterfall sync. We can
remove this hack tomorrow if svn load is still OK.
TBR=cmp
Review URL: https://chromiumcodereview.appspot.com/17468009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@207277 0039d316-1c4b-4281-b951-d872f2087c98
SVN traps SIGINT and attempts to clean itself up, but this results in hangs
waiting for TCP. This patch does two things: daemonizes worker threads so they
are culled when the main thread dies (is ctrl-C'd) and keeps track of spawned
subprocesses to kill any remaining ones when the main program is ctrl-C'd.
A user ctrl-C'ing gclient has to manually terminate hung SVN processes, so this
introduces no extra data loss or hazard. stracing a hung SVN process shows that
it is indeed hanging on TCP reads after receiving a SIGINT, implying there is an
underlying but in the SVN binary.
Review URL: https://chromiumcodereview.appspot.com/14759006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@198205 0039d316-1c4b-4281-b951-d872f2087c98
The 'RemoveDirectory()' function in gclient_utils is deprecated and
rmtree() should be used instead for consistency.
This patch modifies all clients in depot_tools to use rmtree() instead
and removes the RemoveDirectory function.
+ The SVNWrapperTestCase.testRevertNoDotSvn() mocking
expectation has been slightly changed. This was required
because the test invokes code that used to call
gclient_utils.RemoveDirectory() directly, while only
gclient_utils.rmtree() was mocked.
BUG=NONE
R=maruel@chromium.org, ilevy@chromium.org
TEST=manually run gclient_utils_test / gclient_smoketest / scm_unittest / gclient_scm_test
Review URL: https://chromiumcodereview.appspot.com/14134010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@196133 0039d316-1c4b-4281-b951-d872f2087c98
Here's some background why we need this:
We discovered that google code defines the timestamp of a revision to be the time when a commit was started
rather than when it was finished (apache subversion takes the timestamp when the commit transaction is finished).
This can result in a situation where revision R(i-1) has a higher timestamp than Ri.
See bug: https://code.google.com/p/support/issues/detail?id=30419
When using 'gclient --transitive' we effectively do date-based checkouts.
If a parent has a dependency (without a ...@revision) and that dependency lives in the same repository as the parent does
we'd like to checkout the exact same revision as the parent (if we do a date-based checkout as we do now the google code
bug can result in a situation where we don't get the same revision).
Review URL: https://chromiumcodereview.appspot.com/13814012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@194852 0039d316-1c4b-4281-b951-d872f2087c98
Some SVN-only trybots get errors when running gclient revert with this
change. I assume it's because the flag is not present for the gclient
revert command, and that the code checking if options.with_branch_heads
is set is common to both revert and sync.
BUG=none
TEST=reverted locally and gclient revert worked again.
Review URL: https://chromiumcodereview.appspot.com/12616006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@186967 0039d316-1c4b-4281-b951-d872f2087c98
The "branch-heads" are refspecs created from upstream svn branches. This flag
is mostly needed for official builders, and adds about 1/2GB of data to the
checkout, so not currently enabling by default.
Review URL: https://codereview.chromium.org/12546017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@186598 0039d316-1c4b-4281-b951-d872f2087c98
Support 'git grep <match> <branch>' type searches in gclient grep.
These searches had a slightly different output so needed more
tweeking to the regex. I cleaned it up a bit too.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11316213
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@170039 0039d316-1c4b-4281-b951-d872f2087c98
- This will be used by navabi to have his gclient recurse
command print the current repo.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11308115
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@169812 0039d316-1c4b-4281-b951-d872f2087c98
This allows specificying that a target OS list should override the base
list instead of adding to it. This is useful for iOS builds, where the
desired behavior is to pull only the iOS deps, not the union of the Mac
and iOS deps.
Review URL: https://chromiumcodereview.appspot.com/11363036
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@165671 0039d316-1c4b-4281-b951-d872f2087c98
This will let DEPS files specify a higher local bound for recursion.
Note there is an edge case where a dependency can be initially pulled
in with a lower recursion level and then gets ignored subsequently
when a lower level DEPS overrides it's inclusion level and then
includes the same dependency. I do not solve this case, as
I am intending to use this syntax for top level deps files.
BUG=155780
Review URL: https://chromiumcodereview.appspot.com/11146032
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@162464 0039d316-1c4b-4281-b951-d872f2087c98
- On high core, fast machines, jobs=8 is does not offer good
parallelization. Switch to use number of local cpus.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11140019
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@162072 0039d316-1c4b-4281-b951-d872f2087c98
Let's say there's 2 solutions, solA and solB, both fetching the dependency depA
with different urls, solA has 'depA': 'http://foo/bar' and solB has 'depA':
From('depB', 'bar').
This case used to not work, because LateOverride() was not called in
verify_validity(), so it was not comparing the resolved urls but instead what is
textually specified in both DEPS files.
Also because LateOverride() was not called, verify_validity() was comparing the
original urls even if custom_deps is specified in the solution(s).
Finally, allow one to mismatch the other IFF one is None.
R=szager@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11088023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@161228 0039d316-1c4b-4281-b951-d872f2087c98
This simplifies some of the git-submodule stuff. For example, deps2hooks.bat
can be changed from this:
@echo off
setlocal
set TMP_GCLIENT_FILE=.tmp.gclient
echo solutions=[{"name":".","url":None,"deps_file":".DEPS.git"}] > %TMP_GCLIENT_FILE%
FOR /F "delims=" %%a in ('call gclient hookinfo --gclientfile=%TMP_GCLIENT_FILE%') do set HOOKINFO=%%a
set HOOKINFO="call %HOOKINFO:;= & call%"
cd ..
cmd /c %HOOKINFO%
del /F src\%TMP_GCLIENT_FILE%
... to this:
@echo off
setlocal
call gclient runhooks --spec="solutions=[{'name':'src','url':None,'deps_file':'.DEPS.git'}]"
Review URL: https://chromiumcodereview.appspot.com/10835045
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@149223 0039d316-1c4b-4281-b951-d872f2087c98
Cygwin used to have a bug where the number of lock was very limited, causing it
to crash when --jobs was used with a value different than 1. This bug seems to
have been fixed a few months ago as I can't reproduce the anymore after
extensive usage.
R=cmp@chromium.org
BUG=
TEST=
Review URL: https://chromiumcodereview.appspot.com/10689194
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@146798 0039d316-1c4b-4281-b951-d872f2087c98
into stdout and creating the GitDiffFilterer for git repository
BUG=125894
TEST=
Review URL: http://codereview.chromium.org/10317002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@135324 0039d316-1c4b-4281-b951-d872f2087c98
This ability is useful for versions of Chromium (i.e. Android) which want to
cross-compile to another platform that has a fixed set of custom dependencies.
Review URL: http://codereview.chromium.org/10127004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@134280 0039d316-1c4b-4281-b951-d872f2087c98
Previous attempts inside the buildbot config missed some places.
This just does it directly in gclient.
Review carefully, this is a depot_tools change.
BUG=None
TEST=None
R=maruel@chromium.org,dschuff@chromium.org
Review URL: http://codereview.chromium.org/10116033
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@132997 0039d316-1c4b-4281-b951-d872f2087c98
This permits running git grep over the dependencies. With --ignore, gclient
recurse continues even if nothing is found in a dependency. Without this flag,
git grep returns 1 when nothing is found, causing gclient to stop processing the
remaining dependencies.
TBR=nsylvain@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/9600010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124914 0039d316-1c4b-4281-b951-d872f2087c98
Support --jobs in 'fetch' and 'recurse' commands.
BUG=115840
TEST=
Review URL: http://codereview.chromium.org/9560010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124657 0039d316-1c4b-4281-b951-d872f2087c98
The fetch command is simply an alias for:
gclient recurse -s git git fetch origin
Rework 'recurse' command to use a WorkQueue so that --jobs is supported.
BUG=115840
TEST=
Review URL: http://codereview.chromium.org/9463044
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@124334 0039d316-1c4b-4281-b951-d872f2087c98
An example use case is github over ssh at a pinned revision.
R=petermayo@chromium.org
BUG=115202
TEST=
Review URL: http://codereview.chromium.org/9443025
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@123325 0039d316-1c4b-4281-b951-d872f2087c98
This is required to avoid the need to clobber the bots when moving a directory
to deps/. Currently, the directory in question is likely to remain in the
working copy, despite having been removed, due to the presence of untracked
files. This causes the checkout from deps/ to fail.
With this change, when both --reset and --delete_unversioned_trees are
specified, the the directory in question will be removed from the working copy,
thereby allowing the copy in deps/ to be checked out correctly.
Note that untracked directories which are explicitly ignored (ie in .gitignore
or svn:ignore) will not be removed.
Note that this was previously landed in http://codereview.chromium.org/9348054
but reverted due to problems with symlinks in the chromeos build.
BUG=112887, chromium-os:20759
Review URL: http://codereview.chromium.org/9404014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@122300 0039d316-1c4b-4281-b951-d872f2087c98
This is required to avoid the need to clobber the bots when moving a directory
to deps/. Currently, the directory in question is likely to remain in the
working copy, despite having been removed, due to the presence of untracked
files. This causes the checkout from deps/ to fail.
With this change, when both --force and --delete_unversioned_trees are
specified, the the directory in question will be removed from the working copy,
thereby allowing the copy in deps/ to be checked out correctly.
Note that untracked directories which are explicitly ignored (ie in .gitignore
or svn:ignore) will not be removed.
BUG=112887,chromium-os:20759
Review URL: http://codereview.chromium.org/9348054
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@121986 0039d316-1c4b-4281-b951-d872f2087c98
R=maruel@chromium.org
TEST=Configure a checkout using the NewGitWorkflow with a safesync_url and everything works (though possibly with a really long git svn fetch time).
BUG=106015
Review URL: http://codereview.chromium.org/8382030
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@115011 0039d316-1c4b-4281-b951-d872f2087c98
isatty() wasn't defined on Windows, causing it to crash.
R=dpranke@chromium.org
BUG=
TEST=manually gclient sync on windows
Review URL: http://codereview.chromium.org/8371006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106865 0039d316-1c4b-4281-b951-d872f2087c98
For now only affect gclient help to figure out if there is any issue with it.
Because of the way it's done, there is no way to disable coloring, which could
be annoying when redirected. This needs to be taken in consideration.
R=dpranke@chromium.org
TEST=manually tested on linux, cygwin and windows, both run manually and
redirected to a text file.
BUG=
Review URL: http://codereview.chromium.org/8365001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106708 0039d316-1c4b-4281-b951-d872f2087c98
It's not supported at the moment and it's misleading.
Support for it should be added eventually.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8357014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106351 0039d316-1c4b-4281-b951-d872f2087c98
Otherwise it breaks naked --revision usage. One example issue is linux_chromeos
applying the revision to cros_deps instead of src.
TBR=dpranke@chromium.org
BUG=100335
TEST=
Review URL: http://codereview.chromium.org/8298002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105522 0039d316-1c4b-4281-b951-d872f2087c98
It will help catch issues like the one we are getting on linux_chromeos where
the ordering of the solution is now by name, causing the default solution to be
used for --revision being the wrong one.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8276020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105387 0039d316-1c4b-4281-b951-d872f2087c98
It wasn't broken. The tree collapsed around the same time for unrelated reason.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8227036
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105090 0039d316-1c4b-4281-b951-d872f2087c98
I'm not sure if it's causing the tree blow up but I'm not taking any chance.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8228017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104956 0039d316-1c4b-4281-b951-d872f2087c98
This fixes GClientSmokeBoth.testMultiSolutionsJobs flakiness by having
consistent ordering.
Now no out of thread modification is ever done, which result in much saner code.
R=dpranke@chromium.org
BUG=60725
TEST=tested manually with gclient sync --jobs 100 on chrome and with
gclient_smotetest.py
Review URL: http://codereview.chromium.org/8174014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104920 0039d316-1c4b-4281-b951-d872f2087c98
This means parsing the DEPS file all the time preemptively in case it's needed
for a From() reference. This could break certain checkout, we'll see.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8143022
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104817 0039d316-1c4b-4281-b951-d872f2087c98
Also, add more locking to properties.
Now most mutations are contained. Will now figure out a way to remove the
ParseDepsFile() from an unrelated thread.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8142030
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104778 0039d316-1c4b-4281-b951-d872f2087c98
This makes coherency checks more consistent for DEPS vs .gclient.
It's still not thread safe but the mutation code paths are easier to follow.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135019
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104772 0039d316-1c4b-4281-b951-d872f2087c98
... On my quest to make that stuff thread safe.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104033 0039d316-1c4b-4281-b951-d872f2087c98
While trying to fix all the multithreading issues...
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104026 0039d316-1c4b-4281-b951-d872f2087c98
Helps figuring out what is mutable and what is not.
The goal is to shorten the class Dependency to reduce places where mutations can occur.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8122005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@103782 0039d316-1c4b-4281-b951-d872f2087c98
Merge _FindDependencies() back into __init__() now that __init__() is somewhat smaller.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8059011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@103053 0039d316-1c4b-4281-b951-d872f2087c98
First, it reduces the class size, which is too large at the moment.
Second, it makes clear what is 'static' for the livetime of the instance life.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8038015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@102766 0039d316-1c4b-4281-b951-d872f2087c98
The dependencies will continue to be managed. A new flag is present in the .gclient file to control this behavior.
Review URL: http://codereview.chromium.org/7918027
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@102002 0039d316-1c4b-4281-b951-d872f2087c98
Make Dependency.file_list a tuple so it can't be modified.
Move the guesswork of file_list at the right place.
Remove tree() since it's unnecessary.
R=dpranke@chromium.org
BUG=
TEST=all smoke tests still pass
Review URL: http://codereview.chromium.org/7922011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101951 0039d316-1c4b-4281-b951-d872f2087c98
I had forgot one access to _requirements. Now that tests are run, this change is
safer to commit.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7909012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101856 0039d316-1c4b-4281-b951-d872f2087c98
I need to add real thread locking to some Dependency methods. That was long
needed anyway.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7891053
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101100 0039d316-1c4b-4281-b951-d872f2087c98
I've fixed the From() issue that was causing it to hangs. We'll see if anything
pops up again.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7885028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101084 0039d316-1c4b-4281-b951-d872f2087c98
This would result in a hang during gclient sync.
Includes regression test.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7885008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101070 0039d316-1c4b-4281-b951-d872f2087c98
It should be fine now that requirements with /@ works fine.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7782034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@100162 0039d316-1c4b-4281-b951-d872f2087c98
It breaks upstream webkit and nacl. I'll make then both use --jobs 1 before
relanding.
TBR=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7833051
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@99761 0039d316-1c4b-4281-b951-d872f2087c98
Verify correctness with a unit test written explicitly to verify all corner cases.
BUG=60725
TEST=new unit test
Review URL: http://codereview.chromium.org/6598087
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@99234 0039d316-1c4b-4281-b951-d872f2087c98
'Cause linux2 is so last decade.
TBR=evan@chromium.org
BUG=85845
TEST=
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@88788 0039d316-1c4b-4281-b951-d872f2087c98
url is pointing to a git repo, the name of the directory created on disk will not
contain the .git suffix.
Review URL: http://codereview.chromium.org/7104009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@87492 0039d316-1c4b-4281-b951-d872f2087c98
When specifying a revision (with -r) propagates the date of the revision to its children.
Review URL: http://codereview.chromium.org/6873110
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@83313 0039d316-1c4b-4281-b951-d872f2087c98
Make sure every site that catches gclient_utils.Error also catch
subprocess2.CalledProcessError.
BUG=
TEST=
Review URL: http://codereview.chromium.org/6691034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80339 0039d316-1c4b-4281-b951-d872f2087c98
Reapply r79006 with fixes for ubuntu 10.4.
The only change is in fix_default_encoding() to trap an exception
locale.getlocale() may throw.
R=dpranke@chromium.org
BUG=none
TEST=unit test
Review URL: http://codereview.chromium.org/6721029
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79144 0039d316-1c4b-4281-b951-d872f2087c98
It is causing exception for some users on ubuntu 10.4 with:
"category LC_ALL is not supported"
TBR=dpranke
Review URL: http://codereview.chromium.org/6717025
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79012 0039d316-1c4b-4281-b951-d872f2087c98
With this code in action, I can successfully print arabic and chinese on linux
and cygwin. It fails to print chinese on my Mac but prints arabic. It prints '?'
on Windows console but it *doesn't throw* which is the biggest improvement here.
It was particularly a problem on windows because WindowsError's description text
is in the current ANSI code page so it failed to print if the text was not pure
ASCII, like error message when using Windows' French UI.
R=dpranke@chromium.org
BUG=none
TEST=manual unit test ./tests/fix_encoding_test.py and it shouldn't throw
Review URL: http://codereview.chromium.org/6676090
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79006 0039d316-1c4b-4281-b951-d872f2087c98
Don't block execution.
Python 2.4 has been untested for a while and may break unexpectedly.
TEST=manual testing
BUG=none
Review URL: http://codereview.chromium.org/5110009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@66870 0039d316-1c4b-4281-b951-d872f2087c98
Make -D -f combination more effective by deleting directories even if there is
modified files.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/4803001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@65845 0039d316-1c4b-4281-b951-d872f2087c98
Most of them are style issues or variable aliasing.
TEST=Can almost enable pylint warnings
BUG=none
Review URL: http://codereview.chromium.org/4360002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64908 0039d316-1c4b-4281-b951-d872f2087c98
Previously, solution's properties would be ignored if there was no DEPS file in
source control. Change the behavior to evaluate the properties anyway.
TEST=new smoke test
BUG=none
Review URL: http://codereview.chromium.org/4154004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64232 0039d316-1c4b-4281-b951-d872f2087c98
Prints a meaningful message instead of throwing an exception.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3828010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62920 0039d316-1c4b-4281-b951-d872f2087c98
This occurs when gclient compares the cached deps in .gclient_entries
to the new DEPS and tries to construct an SCM for any cached entries
which no longer exist. In this situation gclient doesn't account for
the case where previous entries may have been ignored by setting their
url to None via the custom_deps section in .gclient.
Contributed by Jay Soffian.
Review URL: http://codereview.chromium.org/3769002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62540 0039d316-1c4b-4281-b951-d872f2087c98
sys.stdout.isatty() is True even on buildbot.
Otherwise, logs can't be untangled when using --jobs.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3596006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@61552 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code saner to understand.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3420014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3398008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
The end goal is to remove options.stdout, to remove a lot of bookkeeping.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3418014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
It hoses the chromium slaves on build.webkit.org.
It was fun while it lasted.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3350018
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59146 0039d316-1c4b-4281-b951-d872f2087c98
It seems sys.stdout.isatty() is not a good signal on the build slaves.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59131 0039d316-1c4b-4281-b951-d872f2087c98
This way engineers can still enjoy the perf benefit while someone fixes the
buildbot stdio parsing issue.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3334017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59127 0039d316-1c4b-4281-b951-d872f2087c98
This is because buildbot can't parse gclient output with parallel checkout,
causing got_revision to not be set, causing some scripts to fail.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3325023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59121 0039d316-1c4b-4281-b951-d872f2087c98
Simplify ExecutionQueue.run() by moving code into utility functions.
Reduce the amount of code in WorkerThread.run() to improve reliability.
Don't trap exceptions in single-threaded usage!
BUG=54084
TEST=none
Review URL: http://codereview.chromium.org/3336015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342024
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
pylint is useful to catch errors before they go live so fix little annoyances.
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3372001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58817 0039d316-1c4b-4281-b951-d872f2087c98
This way the function is much more similar to subprocess.call(). Further changes
will be done, to be able to convert all the function, to be able to parallelize
output without corrupting it.
Used pylint to verify call graph correctness, causing some other unrelated
changes.
TEST=unit tests
BUG=none
Review URL: http://codereview.chromium.org/3117039
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57369 0039d316-1c4b-4281-b951-d872f2087c98
It will eventually replace 'print_messages' and be used to annotate printed
lines when using --jobs.
Review URL: http://codereview.chromium.org/3174020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57092 0039d316-1c4b-4281-b951-d872f2087c98
Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.
gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.
--jobs=1 is single threaded as before, albeit with a different code path.
Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.
TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.
---
Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.
runs with -f runs with -m without -f nor -m
args real user real user real user
-j 12 20.59s 18.00s 5.64s 7.95s 5.86s 8.10s
#1 1m05.26s 20.02s 5.20s 7.94s 5.10s 8.09s
22.79s 18.17s
-j 1 #2 1m47.00s 16.72s 9.69s 5.72s 12.35s 5.96s
1m31.28s 17.06s 9.54s 5.85s 10.51s 6.20s
1m31.79s 16.39s
before #3 1m30.94s 16.74s 9.77s 5.83s 10.45s 5.77s
1m30.17s 17.30s 10.36s 5.68s 10.16s 5.88s
hook #4 8.52s 7.93s
8.73s 8.13s
#1 This particular run synched to r56023, a webkit roll updating layout tests.
It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
manually to compare. Hooks are still run in a single thread. I didn't rest
'gclient runhooks'.
I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.
Review URL: http://codereview.chromium.org/3135014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
I've never known why there were added in the first place
Review URL: http://codereview.chromium.org/3155008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55896 0039d316-1c4b-4281-b951-d872f2087c98
Instead of keeping the reference information in the parent, note for each Dependency if it should be processed with self.should_process. I had to hack a bit with the hooks to also enforce recursion_limit() to keep the old behavior, otherwise From() could cause hooks to run that weren't run before.
BUG=50015
TEST=hooks are run properly. Tested with webkit and pagespeed and buildbot master, fixes running the same checkout multiple times.
Review URL: http://codereview.chromium.org/3124017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55895 0039d316-1c4b-4281-b951-d872f2087c98
ExecutionQueue takes care of reordering execution depending on each Dependency
requirements. There is no need to sort anymore.
ExecutionQueue also manages the Progress instance and is multithread safe, in preparation to parallel checkout.
Review URL: http://codereview.chromium.org/3112002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55882 0039d316-1c4b-4281-b951-d872f2087c98
gclient revinfo alone will now not return the actual checked out revisions but simply what's in the .gclient&DEPS.
BUG=none
TEST=add a --snapshot test to make sure it is not broken
Review URL: http://codereview.chromium.org/3130007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55760 0039d316-1c4b-4281-b951-d872f2087c98
Add Dependency.processed and .hooks_ran to ease debugging.
Accept when more than one reference exist for FromImpl(), it may happen as it is the case for pagespeed.
Not much behavior change.
Review URL: http://codereview.chromium.org/3126003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55575 0039d316-1c4b-4281-b951-d872f2087c98
It would modify self.parsed_url even when it's called for a From() keyword
Review URL: http://codereview.chromium.org/3171001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55574 0039d316-1c4b-4281-b951-d872f2087c98
Also rename an argument to better fit reality.
No functional change
Review URL: http://codereview.chromium.org/3108001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55559 0039d316-1c4b-4281-b951-d872f2087c98
This was the behavior before the refactor.
TEST=sync parfait and hooks should be run when necessary.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2819062
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53430 0039d316-1c4b-4281-b951-d872f2087c98
This issue is exhibited by pagespeed.
Also a solution is always a direct dependency by definition.
TEST=none
TBR=bradnelson
Review URL: http://codereview.chromium.org/3036013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53397 0039d316-1c4b-4281-b951-d872f2087c98
No behavior change.
TEST=all test still pass
TBR=bradnelson
Review URL: http://codereview.chromium.org/3005026
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53392 0039d316-1c4b-4281-b951-d872f2087c98
PageSpeed is an example of such ordering dependency.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2809072
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53377 0039d316-1c4b-4281-b951-d872f2087c98
Bring some OOP and sanity to gclient.py.
- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.
TEST=all test pass
The main difference with r51760 is that new smoke tests have been added since which replicates how webkit checkouts with gclient.
Review URL: http://codereview.chromium.org/2917009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53205 0039d316-1c4b-4281-b951-d872f2087c98
Chromium OS continuous builds are failing because of the following error.
Traceback (most recent call last):
File "/b/depot_tools/gclient.py", line 1212, in <module>
sys.exit(Main(sys.argv[1:]))
File "/b/depot_tools/gclient.py", line 1202, in Main
return command(parser, argv[1:])
File "/b/depot_tools/gclient.py", line 1122, in CMDrevinfo
client.PrintRevInfo()
File "/b/depot_tools/gclient.py", line 776, in PrintRevInfo
url = entries.pop(name)
KeyError: 'chromiumos'
This change will fix the error.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/2892007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52180 0039d316-1c4b-4281-b951-d872f2087c98
This is being abused by WebKit.
TEST=new smoke test
Review URL: http://codereview.chromium.org/2923005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52096 0039d316-1c4b-4281-b951-d872f2087c98
This case covers the webkit gclient usage.
TEST=improved smoke test
Review URL: http://codereview.chromium.org/2968005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52092 0039d316-1c4b-4281-b951-d872f2087c98
Add the fix for FileImpl() plus unit tests.
TEST=more tests
Review URL: http://codereview.chromium.org/2808048
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52005 0039d316-1c4b-4281-b951-d872f2087c98
iannucci@chromium.org
rwalker@codeaurora.org
nodir@chromium.org
sergiyb@chromium.org
tandrii@chromium.org
thakis@chromium.org
szager@chromium.org
vapier@chromium.org
agable@chromium.org
stip@chromium.org
e.hakkinen@samsung.com
sbc@chromium.org
dnj@chromium.org
primiano@chromium.org
mmoss@chromium.org
iannucci@chromium.org
szager@chromium.org
dyen@chromium.org
primiano@chromium.org
cmp@chromium.org
smut@google.com
dnj@chromium.org
hinoka@chromium.org
borenet@google.com
scr@chromium.org
rnk@chromium.org
hinoka@google.com
tnagel@chromium.org
mmoss@chromium.org
rdsmith@chromium.org
thakis@chromium.org
bratell@opera.com
scottmg@chromium.org
maruel@chromium.org
xusydoc@chromium.org
ilevy@chromium.org
petermayo@chromium.org
digit@chromium.org
kustermann@google.com
jochen@chromium.org
kjellander@chromium.org
bcwhite@chromium.org
sivachandra@chromium.org
stuartmorgan@chromium.org
dbeam@chromium.org
haitao.feng@intel.com
peter@chromium.org
bradnelson@google.com
szager@google.com
davidbarr@chromium.org
steveblock@chromium.org
phajdan.jr@chromium.org
bauerb@chromium.org
nsylvain@google.com
floitsch@google.com
thomasvl@chromium.org
joi@chromium.org
mazda@chromium.org