This updates some infrastructure to make it easy to get at the platform-specific build tools directories.
Review URL: https://codereview.chromium.org/341533006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@279132 0039d316-1c4b-4281-b951-d872f2087c98
Features:
- Non-verbose output is now limited to a one-line progress
indicator.
- Verbose output is now collated per subprocess. As soon as a
subprocess finishes, its full output is dumped to terminal.
- Verbose output is prefixed with timestamps representing elapsed
time since the beginning of the gclient invocation.
- git progress indicators ("Receiving objects", etc.) are limited to
one line every 10 seconds.
- In both verbose and non-verbose mode, if a failure occurs, the
full output of the failed update operation is dumped to terminal
just before exit.
- In the event that updates are progressing, but slowly,
"Still working" messages will be printed periodically, to pacify
users and buildbots.
BUG=
R=hinoka@google.com
Review URL: https://codereview.chromium.org/227163002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@262500 0039d316-1c4b-4281-b951-d872f2087c98
Instead of having custom logic for dealing with cache directories, use
git_cache.py to populate caches.
Also fixes a bug in git_cache.py where it was looking for lockfiles in cwd rather than the cache dir.
Other changes:
* _Run now returns output.
* Always print to stdout in CheckCallAndFilterOutput, even if it gets a carriage return. This is done because git progress report are carriage returns and not newlines and we don't want everything on the same line and not strip out the CRs.
* Removed members changed tests, its not very useful to know a new import is added.
BUG=339171
Review URL: https://codereview.chromium.org/180243006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@254248 0039d316-1c4b-4281-b951-d872f2087c98
This reverts commit 6ff5448534.
'fetch --nohooks chromium' is broken with this change.
BUG=
Review URL: https://codereview.chromium.org/177003023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@253640 0039d316-1c4b-4281-b951-d872f2087c98
The git cache command is a central place to manage a machine's git cache.
It provides two subcommands:
* populate -- creates or updates the cache of a given repository
* exists -- prints the path to the cache of a repo, if it exists
Gclient, deps2git, bot_update, and any other tools that touch the cache will
be able to use this command to make sure that everyone is interacting with
the cache in the same way.
R=hinoka@chromium.org, iannucci@chromium.org
BUG=339168
Review URL: https://codereview.chromium.org/164823002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@253007 0039d316-1c4b-4281-b951-d872f2087c98
as part of the Chromium checkout.
This follows the approach used by gn.
Changes include:
- in-the-PATH clang-format trampoline scripts
- clang_format.py, which finds clang-format binaries inside of Chrome
- Hook 'git cl format' to the new binaries and scripts
- Rearrange some code, for reuse between clang_format.py and gn.py
BUG=240309
TEST=presubmits (one failure on mac, but it fails on a clean checkout too)
Review URL: https://codereview.chromium.org/134313007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@245074 0039d316-1c4b-4281-b951-d872f2087c98
Not all Linux machines have vim preinstalled (example: Ubuntu 12.04).
All machines have vi though. And it's actually vim but stripped down.
BUG=
Review URL: https://codereview.chromium.org/101823005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@239488 0039d316-1c4b-4281-b951-d872f2087c98
sleep() is in time, not sys.
BUG=
R=maruel@chromium.org,szager@chromium.org,cmp@chromium.org
Review URL: https://codereview.chromium.org/34483004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@230108 0039d316-1c4b-4281-b951-d872f2087c98
Add retry for all git operations that speak to a remote. This
should smooth out issues with the git/gerrit-on-borg service.
Review URL: https://codereview.chromium.org/26234004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@229219 0039d316-1c4b-4281-b951-d872f2087c98
For example, my id is 'kangil.han', but it fails during parsing.
So this patch fixes it.
Review URL: https://codereview.chromium.org/27484002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@228937 0039d316-1c4b-4281-b951-d872f2087c98
This solves a problem with "os.rename" sometimes failing with an
exception after cloning a dependency to a temporary directory. It's
possible the dying git processes still hold a handle to the directory.
Since gclient delete the temporary directory regardless of the success
of the process, it results in a lot of GB downloaded for nothing.
SafeRename solves this by retrying a few times if the renaming fails,
sleeping one second every time to get other processes the time to
release their lock on the directory. It gives up retrying after 15 times.
BUG=
R=maruel@chromium.org
Review URL: https://chromiumcodereview.appspot.com/23875041
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@224372 0039d316-1c4b-4281-b951-d872f2087c98
So that if the process dies in the middle, the files are easier to spot and
delete.
R=iannucci@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/19498004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@213081 0039d316-1c4b-4281-b951-d872f2087c98
We can use this to evaluate the usefulness of making hooks run in parallel.
Review URL: https://chromiumcodereview.appspot.com/18851005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@211446 0039d316-1c4b-4281-b951-d872f2087c98
SVN traps SIGINT and attempts to clean itself up, but this results in hangs
waiting for TCP. This patch does two things: daemonizes worker threads so they
are culled when the main thread dies (is ctrl-C'd) and keeps track of spawned
subprocesses to kill any remaining ones when the main program is ctrl-C'd.
A user ctrl-C'ing gclient has to manually terminate hung SVN processes, so this
introduces no extra data loss or hazard. stracing a hung SVN process shows that
it is indeed hanging on TCP reads after receiving a SIGINT, implying there is an
underlying but in the SVN binary.
Review URL: https://chromiumcodereview.appspot.com/14759006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@198205 0039d316-1c4b-4281-b951-d872f2087c98
It is somewhat surprising when git-cl, which acts as a git subcommand, launches
a different editor. In particular, git has a config option (core.editor) which
specifies the editor that should be used. Since we already respect $GIT_EDITOR,
it makes sense for git-cl to respect core.editor and $VISUAL as well.
R=maruel@chromium.org
BUG=237504
Review URL: https://chromiumcodereview.appspot.com/14854003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@198101 0039d316-1c4b-4281-b951-d872f2087c98
The 'RemoveDirectory()' function in gclient_utils is deprecated and
rmtree() should be used instead for consistency.
This patch modifies all clients in depot_tools to use rmtree() instead
and removes the RemoveDirectory function.
+ The SVNWrapperTestCase.testRevertNoDotSvn() mocking
expectation has been slightly changed. This was required
because the test invokes code that used to call
gclient_utils.RemoveDirectory() directly, while only
gclient_utils.rmtree() was mocked.
BUG=NONE
R=maruel@chromium.org, ilevy@chromium.org
TEST=manually run gclient_utils_test / gclient_smoketest / scm_unittest / gclient_scm_test
Review URL: https://chromiumcodereview.appspot.com/14134010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@196133 0039d316-1c4b-4281-b951-d872f2087c98
This is needed because the current implementation commonly fails on Windows due to "directory not empty" errors. Using rd is more reliable.
Review URL: https://chromiumcodereview.appspot.com/13581005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@194932 0039d316-1c4b-4281-b951-d872f2087c98
- On high core, fast machines, jobs=8 is does not offer good
parallelization. Switch to use number of local cpus.
R=maruel@chromium.org
BUG=
Review URL: https://chromiumcodereview.appspot.com/11140019
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@162072 0039d316-1c4b-4281-b951-d872f2087c98
Since open() and codecs.open() have different behavior, and codecs.open()
misbehaves on python 2.6, switch to open() with manual str decoding.
TBR=cmp@chromium.org
BUG=
TEST=I failed to reproduce, it's a shot in the dark
Review URL: https://chromiumcodereview.appspot.com/10698089
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@145396 0039d316-1c4b-4281-b951-d872f2087c98
Otherwise, the files are opened in 'whatever happens to be the current encoding'
which is highly system-dependent. Even if some files are not encoded with utf-8,
the status quo is even worse. So it's worth trying out.
TBR=cmp@chromium.org
BUG=
TEST=
Review URL: https://chromiumcodereview.appspot.com/10697036
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@145306 0039d316-1c4b-4281-b951-d872f2087c98
Enable it for git-cl and gcl.
R=nsylvain@chromium.org
BUG=107838
TEST=New connections go through https://
Review URL: http://codereview.chromium.org/9214004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@117857 0039d316-1c4b-4281-b951-d872f2087c98
So any modification to the defaults is coded at only one place.
TBR=nsylvain@chromium.org
BUG=107838
TEST=
Review URL: http://codereview.chromium.org/9214002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@117853 0039d316-1c4b-4281-b951-d872f2087c98
Windows users wouldn't get the same behavior on git cl vs gcl.
Improve automatic CRLF<->LF conversion, some gcl users would be \n repeated in
their description depending on the editor used.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8360007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@107106 0039d316-1c4b-4281-b951-d872f2087c98
directory doesn't exist, on fresh checkout and --jobs >1.
It happens in the case where there is 3 dependencies:
a
a/b/c/d
a/b/c/e
'a' is processed first. Then 'a/b/c/d' and 'a/b/c/e' are fired simultaneously.
If both are not present, both svn checkout tries to mkdir b and b/c and a race
condition occurs between their call for isdir() and mkdir().
This works around the issue by creating the intermediary directories ourself
before calling out svn and managing the error ourself.
TBR=dpranke@chromium.org
BUG=
TEST=fresh checkout shouldn't be flaky
Review URL: http://codereview.chromium.org/8359018
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106636 0039d316-1c4b-4281-b951-d872f2087c98
Reuse a common lock to reduce the number of locks created. Poor cygwin users.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8353010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106494 0039d316-1c4b-4281-b951-d872f2087c98
Fix a problem introduced in r105229.
R=cmp@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8349001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@106089 0039d316-1c4b-4281-b951-d872f2087c98
This is really bad but the lock is never held for a long time so it 'should' work.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8253008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@105195 0039d316-1c4b-4281-b951-d872f2087c98
This fixes GClientSmokeBoth.testMultiSolutionsJobs flakiness by having
consistent ordering.
Now no out of thread modification is ever done, which result in much saner code.
R=dpranke@chromium.org
BUG=60725
TEST=tested manually with gclient sync --jobs 100 on chrome and with
gclient_smotetest.py
Review URL: http://codereview.chromium.org/8174014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104920 0039d316-1c4b-4281-b951-d872f2087c98
... On my quest to make that stuff thread safe.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104033 0039d316-1c4b-4281-b951-d872f2087c98
While trying to fix all the multithreading issues...
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/8135007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@104026 0039d316-1c4b-4281-b951-d872f2087c98
I had forgot one access to _requirements. Now that tests are run, this change is
safer to commit.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7909012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101856 0039d316-1c4b-4281-b951-d872f2087c98
This would result in a hang during gclient sync.
Includes regression test.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/7885008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@101070 0039d316-1c4b-4281-b951-d872f2087c98
While using --jobs, empty lines are adding a lot of noise.
An example output would look like:
>gclient sync --jobs=10
1>
3>
4>
6>
5>
7>
5>
5>________ running 'git clone ...'
Review URL: http://codereview.chromium.org/7101012
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@87542 0039d316-1c4b-4281-b951-d872f2087c98
When specifying a revision (with -r) propagates the date of the revision to its children.
Review URL: http://codereview.chromium.org/6873110
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@83313 0039d316-1c4b-4281-b951-d872f2087c98
The reason of this change (beside deleting 30 lines) is to fix pylint on Mac.
For an unknown reason, it was dying here trying to process the minidom import.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/6799021
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80770 0039d316-1c4b-4281-b951-d872f2087c98
It should be fine now that I've fixed python2.5 compatilibity issues in subprocess2.py.
R=dpranke@chromium.org
BUG=
TEST=
Review URL: http://codereview.chromium.org/6759074
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80216 0039d316-1c4b-4281-b951-d872f2087c98
This change moves most of the code into presubmit_support and gclient_utils and just leaves the formatting differences for the messages between the two tools.
Review URL: http://codereview.chromium.org/6719004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79002 0039d316-1c4b-4281-b951-d872f2087c98
pass pylint cleanly.
There should be no functional changes in this patch, although there's at least one minor bug that lint caught.
Review URL: http://codereview.chromium.org/6685023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@78638 0039d316-1c4b-4281-b951-d872f2087c98
It would fail on win32 when these modules are not present.
BUG=75902
TEST=none
R=mhm@chromium.org
Review URL: http://codereview.chromium.org/6683028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@77947 0039d316-1c4b-4281-b951-d872f2087c98
Keep an alias to RemoveDirectory, will be removed in a later change.
TEST=new unit test to test this function
BUG=none
Review URL: http://codereview.chromium.org/6628032
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@77463 0039d316-1c4b-4281-b951-d872f2087c98
When an HTTP exception information is logged, no information is sent by
default. Add an helper function to manually extract interesting information
to append it to the exception type.
Also disable another pylint warning.
TEST=manual test
BUG=none
Review URL: http://codereview.chromium.org/5303002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@67306 0039d316-1c4b-4281-b951-d872f2087c98
Python standard library is not thread safe. Work around it by leaking a bit.
The problem is particularly visible on python 2.4.
TEST=none
BUG=60623
Review URL: http://codereview.chromium.org/4465001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@65048 0039d316-1c4b-4281-b951-d872f2087c98
Most of them are style issues or variable aliasing.
TEST=Can almost enable pylint warnings
BUG=none
Review URL: http://codereview.chromium.org/4360002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64908 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code saner to understand.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3420014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3398008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
The end goal is to remove options.stdout, to remove a lot of bookkeeping.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3418014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
BUG=54084
TEST=updated smoke test and added a new one + stdout untangling
Review URL: http://codereview.chromium.org/3357020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59002 0039d316-1c4b-4281-b951-d872f2087c98
Simplify ExecutionQueue.run() by moving code into utility functions.
Reduce the amount of code in WorkerThread.run() to improve reliability.
Don't trap exceptions in single-threaded usage!
BUG=54084
TEST=none
Review URL: http://codereview.chromium.org/3336015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
Add back support for python 2.4.
Looks like a few slaves haven't been updated yet.
TBR=bradnelson
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3338017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58820 0039d316-1c4b-4281-b951-d872f2087c98
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342024
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
jochen@chromium.org
brettw@chromium.org
hinoka@google.com
scr@chromium.org
szager@chromium.org
borenet@google.com
agable@chromium.org
nick@chromium.org
bratell@opera.com
raphael.kubo.da.costa@intel.com
kangil.han@samsung.com
scottmg@chromium.org
cyrille@nnamrak.org
maruel@chromium.org
ilevy@chromium.org
xusydoc@chromium.org
jbroman@chromium.org
digit@chromium.org
jam@chromium.org
chrisha@chromium.org
mukai@chromium.org
asvitkine@chromium.org
rcui@google.com
szager@google.com
nsylvain@google.com
floitsch@google.com
dpranke@chromium.org
thakis@chromium.org