Verify correctness with a unit test written explicitly to verify all corner cases.
BUG=60725
TEST=new unit test
Review URL: http://codereview.chromium.org/6598087
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@99234 0039d316-1c4b-4281-b951-d872f2087c98
'Cause linux2 is so last decade.
TBR=evan@chromium.org
BUG=85845
TEST=
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@88788 0039d316-1c4b-4281-b951-d872f2087c98
url is pointing to a git repo, the name of the directory created on disk will not
contain the .git suffix.
Review URL: http://codereview.chromium.org/7104009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@87492 0039d316-1c4b-4281-b951-d872f2087c98
When specifying a revision (with -r) propagates the date of the revision to its children.
Review URL: http://codereview.chromium.org/6873110
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@83313 0039d316-1c4b-4281-b951-d872f2087c98
Make sure every site that catches gclient_utils.Error also catch
subprocess2.CalledProcessError.
BUG=
TEST=
Review URL: http://codereview.chromium.org/6691034
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@80339 0039d316-1c4b-4281-b951-d872f2087c98
Reapply r79006 with fixes for ubuntu 10.4.
The only change is in fix_default_encoding() to trap an exception
locale.getlocale() may throw.
R=dpranke@chromium.org
BUG=none
TEST=unit test
Review URL: http://codereview.chromium.org/6721029
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79144 0039d316-1c4b-4281-b951-d872f2087c98
It is causing exception for some users on ubuntu 10.4 with:
"category LC_ALL is not supported"
TBR=dpranke
Review URL: http://codereview.chromium.org/6717025
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79012 0039d316-1c4b-4281-b951-d872f2087c98
With this code in action, I can successfully print arabic and chinese on linux
and cygwin. It fails to print chinese on my Mac but prints arabic. It prints '?'
on Windows console but it *doesn't throw* which is the biggest improvement here.
It was particularly a problem on windows because WindowsError's description text
is in the current ANSI code page so it failed to print if the text was not pure
ASCII, like error message when using Windows' French UI.
R=dpranke@chromium.org
BUG=none
TEST=manual unit test ./tests/fix_encoding_test.py and it shouldn't throw
Review URL: http://codereview.chromium.org/6676090
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@79006 0039d316-1c4b-4281-b951-d872f2087c98
Don't block execution.
Python 2.4 has been untested for a while and may break unexpectedly.
TEST=manual testing
BUG=none
Review URL: http://codereview.chromium.org/5110009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@66870 0039d316-1c4b-4281-b951-d872f2087c98
Make -D -f combination more effective by deleting directories even if there is
modified files.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/4803001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@65845 0039d316-1c4b-4281-b951-d872f2087c98
Most of them are style issues or variable aliasing.
TEST=Can almost enable pylint warnings
BUG=none
Review URL: http://codereview.chromium.org/4360002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64908 0039d316-1c4b-4281-b951-d872f2087c98
Previously, solution's properties would be ignored if there was no DEPS file in
source control. Change the behavior to evaluate the properties anyway.
TEST=new smoke test
BUG=none
Review URL: http://codereview.chromium.org/4154004
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@64232 0039d316-1c4b-4281-b951-d872f2087c98
Prints a meaningful message instead of throwing an exception.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3828010
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62920 0039d316-1c4b-4281-b951-d872f2087c98
This occurs when gclient compares the cached deps in .gclient_entries
to the new DEPS and tries to construct an SCM for any cached entries
which no longer exist. In this situation gclient doesn't account for
the case where previous entries may have been ignored by setting their
url to None via the custom_deps section in .gclient.
Contributed by Jay Soffian.
Review URL: http://codereview.chromium.org/3769002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62540 0039d316-1c4b-4281-b951-d872f2087c98
sys.stdout.isatty() is True even on buildbot.
Otherwise, logs can't be untangled when using --jobs.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3596006
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@61552 0039d316-1c4b-4281-b951-d872f2087c98
This makes the code saner to understand.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3420014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3398008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
The end goal is to remove options.stdout, to remove a lot of bookkeeping.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3418014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
It hoses the chromium slaves on build.webkit.org.
It was fun while it lasted.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3350018
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59146 0039d316-1c4b-4281-b951-d872f2087c98
It seems sys.stdout.isatty() is not a good signal on the build slaves.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342028
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59131 0039d316-1c4b-4281-b951-d872f2087c98
This way engineers can still enjoy the perf benefit while someone fixes the
buildbot stdio parsing issue.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3334017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59127 0039d316-1c4b-4281-b951-d872f2087c98
This is because buildbot can't parse gclient output with parallel checkout,
causing got_revision to not be set, causing some scripts to fail.
TBR=bradnelson
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3325023
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59121 0039d316-1c4b-4281-b951-d872f2087c98
Simplify ExecutionQueue.run() by moving code into utility functions.
Reduce the amount of code in WorkerThread.run() to improve reliability.
Don't trap exceptions in single-threaded usage!
BUG=54084
TEST=none
Review URL: http://codereview.chromium.org/3336015
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/3342024
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
pylint is useful to catch errors before they go live so fix little annoyances.
BUG=none
TEST=none
Review URL: http://codereview.chromium.org/3372001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58817 0039d316-1c4b-4281-b951-d872f2087c98
This way the function is much more similar to subprocess.call(). Further changes
will be done, to be able to convert all the function, to be able to parallelize
output without corrupting it.
Used pylint to verify call graph correctness, causing some other unrelated
changes.
TEST=unit tests
BUG=none
Review URL: http://codereview.chromium.org/3117039
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57369 0039d316-1c4b-4281-b951-d872f2087c98
It will eventually replace 'print_messages' and be used to annotate printed
lines when using --jobs.
Review URL: http://codereview.chromium.org/3174020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57092 0039d316-1c4b-4281-b951-d872f2087c98
Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.
gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.
--jobs=1 is single threaded as before, albeit with a different code path.
Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.
TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.
---
Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.
runs with -f runs with -m without -f nor -m
args real user real user real user
-j 12 20.59s 18.00s 5.64s 7.95s 5.86s 8.10s
#1 1m05.26s 20.02s 5.20s 7.94s 5.10s 8.09s
22.79s 18.17s
-j 1 #2 1m47.00s 16.72s 9.69s 5.72s 12.35s 5.96s
1m31.28s 17.06s 9.54s 5.85s 10.51s 6.20s
1m31.79s 16.39s
before #3 1m30.94s 16.74s 9.77s 5.83s 10.45s 5.77s
1m30.17s 17.30s 10.36s 5.68s 10.16s 5.88s
hook #4 8.52s 7.93s
8.73s 8.13s
#1 This particular run synched to r56023, a webkit roll updating layout tests.
It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
manually to compare. Hooks are still run in a single thread. I didn't rest
'gclient runhooks'.
I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.
Review URL: http://codereview.chromium.org/3135014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
I've never known why there were added in the first place
Review URL: http://codereview.chromium.org/3155008
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55896 0039d316-1c4b-4281-b951-d872f2087c98
Instead of keeping the reference information in the parent, note for each Dependency if it should be processed with self.should_process. I had to hack a bit with the hooks to also enforce recursion_limit() to keep the old behavior, otherwise From() could cause hooks to run that weren't run before.
BUG=50015
TEST=hooks are run properly. Tested with webkit and pagespeed and buildbot master, fixes running the same checkout multiple times.
Review URL: http://codereview.chromium.org/3124017
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55895 0039d316-1c4b-4281-b951-d872f2087c98
ExecutionQueue takes care of reordering execution depending on each Dependency
requirements. There is no need to sort anymore.
ExecutionQueue also manages the Progress instance and is multithread safe, in preparation to parallel checkout.
Review URL: http://codereview.chromium.org/3112002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55882 0039d316-1c4b-4281-b951-d872f2087c98
gclient revinfo alone will now not return the actual checked out revisions but simply what's in the .gclient&DEPS.
BUG=none
TEST=add a --snapshot test to make sure it is not broken
Review URL: http://codereview.chromium.org/3130007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55760 0039d316-1c4b-4281-b951-d872f2087c98
Add Dependency.processed and .hooks_ran to ease debugging.
Accept when more than one reference exist for FromImpl(), it may happen as it is the case for pagespeed.
Not much behavior change.
Review URL: http://codereview.chromium.org/3126003
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55575 0039d316-1c4b-4281-b951-d872f2087c98
It would modify self.parsed_url even when it's called for a From() keyword
Review URL: http://codereview.chromium.org/3171001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55574 0039d316-1c4b-4281-b951-d872f2087c98
Also rename an argument to better fit reality.
No functional change
Review URL: http://codereview.chromium.org/3108001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55559 0039d316-1c4b-4281-b951-d872f2087c98
This was the behavior before the refactor.
TEST=sync parfait and hooks should be run when necessary.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2819062
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53430 0039d316-1c4b-4281-b951-d872f2087c98
This issue is exhibited by pagespeed.
Also a solution is always a direct dependency by definition.
TEST=none
TBR=bradnelson
Review URL: http://codereview.chromium.org/3036013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53397 0039d316-1c4b-4281-b951-d872f2087c98
No behavior change.
TEST=all test still pass
TBR=bradnelson
Review URL: http://codereview.chromium.org/3005026
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53392 0039d316-1c4b-4281-b951-d872f2087c98
PageSpeed is an example of such ordering dependency.
TBR=bradnelson
Review URL: http://codereview.chromium.org/2809072
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53377 0039d316-1c4b-4281-b951-d872f2087c98
Bring some OOP and sanity to gclient.py.
- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.
TEST=all test pass
The main difference with r51760 is that new smoke tests have been added since which replicates how webkit checkouts with gclient.
Review URL: http://codereview.chromium.org/2917009
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53205 0039d316-1c4b-4281-b951-d872f2087c98
Chromium OS continuous builds are failing because of the following error.
Traceback (most recent call last):
File "/b/depot_tools/gclient.py", line 1212, in <module>
sys.exit(Main(sys.argv[1:]))
File "/b/depot_tools/gclient.py", line 1202, in Main
return command(parser, argv[1:])
File "/b/depot_tools/gclient.py", line 1122, in CMDrevinfo
client.PrintRevInfo()
File "/b/depot_tools/gclient.py", line 776, in PrintRevInfo
url = entries.pop(name)
KeyError: 'chromiumos'
This change will fix the error.
TEST=none
BUG=none
Review URL: http://codereview.chromium.org/2892007
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52180 0039d316-1c4b-4281-b951-d872f2087c98
This is being abused by WebKit.
TEST=new smoke test
Review URL: http://codereview.chromium.org/2923005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52096 0039d316-1c4b-4281-b951-d872f2087c98
This case covers the webkit gclient usage.
TEST=improved smoke test
Review URL: http://codereview.chromium.org/2968005
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52092 0039d316-1c4b-4281-b951-d872f2087c98
Add the fix for FileImpl() plus unit tests.
TEST=more tests
Review URL: http://codereview.chromium.org/2808048
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52005 0039d316-1c4b-4281-b951-d872f2087c98
I didn't run the unit test after doing the last minute change and it was indeed broken.
TBR=msb
Review URL: http://codereview.chromium.org/2823040
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51761 0039d316-1c4b-4281-b951-d872f2087c98
- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.
TEST=all test pass
Review URL: http://codereview.chromium.org/2867047
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51760 0039d316-1c4b-4281-b951-d872f2087c98
TEST=none, no big deal. I'll get a breakpad report otherwise anyway. This is solely to *reduce* the number of breakpad stack trace that this is done.
Review URL: http://codereview.chromium.org/2885020
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51739 0039d316-1c4b-4281-b951-d872f2087c98
Remove SUPPORTED_COMMANDS since it is not necessary, change the formating to not introduce an empty line.
Review URL: http://codereview.chromium.org/2836042
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51640 0039d316-1c4b-4281-b951-d872f2087c98
This allows to recurse in all the entries to operate on them.
Review URL: http://codereview.chromium.org/2862039
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@51419 0039d316-1c4b-4281-b951-d872f2087c98
The change is partial to keep this change still correct but readable/reviewable.
Followup changes will further move functions into the Dependency class.
If it was done in one change, it would be unreviewable.
Fix GetScmName() for protocol svn+ssh://.
TEST=didn't break the smoke tests.
Review URL: http://codereview.chromium.org/2837001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@49707 0039d316-1c4b-4281-b951-d872f2087c98
Another step in my quest to make the refactor reviewable.
Review URL: http://codereview.chromium.org/2786014
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@49601 0039d316-1c4b-4281-b951-d872f2087c98
Fix gclient diff & pack to not throw uncatched exceptions when a directory is missing.
Reorder things in gclient.py for easier diff later
Update revinfo help
Add GetScmName(), it will be needed later.
TEST=unit tests
BUG=none
Review URL: http://codereview.chromium.org/2786013
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@49565 0039d316-1c4b-4281-b951-d872f2087c98
Switch "" usage to '' to please Brad. Didn't change gclient.py yet because it
will conflict with the refactor.
TEST=unit tests still pass.
Review URL: http://codereview.chromium.org/2769011
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@49545 0039d316-1c4b-4281-b951-d872f2087c98
Make -r invalid@rev a warning and ignore the argument.
Make -r rev use the Nth solution.
Fix a bug in FakeRepos.assertTree().
TEST=updated the smoke test to reflect the new behavior which should be the most compatible with the current buildbot behavior
BUG=chromium-os:3465
Review URL: http://codereview.chromium.org/2430001
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@48607 0039d316-1c4b-4281-b951-d872f2087c98
Make it work with 'coverage'.
Make FakeRepos initialization lazy. Only generate the repository when needed. It's in part intended for gclient_scm_tests.py but I left it out of this review.
To use:
easy_install coverage
./tests/gclient_smoke.py -c
coverage report
Current coverage with gclient_smoke:
Name Stmts Exec Cover
-----------------------------------------------
gclient 557 416 74%
gclient_scm 496 270 54%
gclient_utils 195 117 60%
scm 439 180 41%
(ignoring irrelevant files)
It's quite good in fact, 74% of gclient.py being executed with the smoke test.
Review URL: http://codereview.chromium.org/2352002
git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@48501 0039d316-1c4b-4281-b951-d872f2087c98
maruel@chromium.org
phajdan.jr@chromium.org
bauerb@chromium.org
nsylvain@google.com
floitsch@google.com
thomasvl@chromium.org
joi@chromium.org
jochen@chromium.org
bradnelson@google.com
mazda@chromium.org
piman@chromium.org