Commit Graph

170 Commits (78cba52586d2b43518159754ce414571c56ad4c0)

Author SHA1 Message Date
maruel@chromium.org 78cba52586 gclient recurse fails if .gclient_entries doesn't exist.
Prints a meaningful message instead of throwing an exception.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3828010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62920 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 04dd7deb6b Fix "No SCM found for url None" error that can occur after update
This occurs when gclient compares the cached deps in .gclient_entries
to the new DEPS and tries to construct an SCM for any cached entries
which no longer exist. In this situation gclient doesn't account for
the case where previous entries may have been ignored by setting their
url to None via the custom_deps section in .gclient.

Contributed by Jay Soffian.

Review URL: http://codereview.chromium.org/3769002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62540 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org ac61023c2c Fix a bug in gclient recurse for git-svn users. Make gclient_utils.CheckCall more versatile.
TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3737001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@62400 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org a116e7d373 Workaround to disable Progress() on buildbot by disabling it on --verbose.
sys.stdout.isatty() is True even on buildbot.
Otherwise, logs can't be untangled when using --jobs.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3596006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@61552 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 77e4ecaf59 Remove cruft from having to do bookeeping of options.stdout.
This makes the code saner to understand.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3420014

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@60050 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 4ed3418912 Directly hook sys.stdout for thread annotated output.
In the next change, we can now remove all the options.stdout bookeeping since
it's not unnecessary.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3398008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59795 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org e0de9cbe0f Remove the class StdoutAnnotated and clones the object instead.
The end goal is to remove options.stdout, to remove a lot of bookkeeping.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3418014

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59792 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org fac9d3d690 Give up on parallel checkout by default.
It hoses the chromium slaves on build.webkit.org.
It was fun while it lasted.

TBR=bradnelson
TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3350018

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59146 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 5f2a196ca6 Change the way to detect build slave.
It seems sys.stdout.isatty() is not a good signal on the build slaves.

TBR=bradnelson
TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3342028

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59131 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org e75e8e683c Add a cheap trick to disable parallel sync on the buildbot until they are fixed.
This way engineers can still enjoy the perf benefit while someone fixes the
buildbot stdio parsing issue.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3334017

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59127 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 0d2bf39ca9 Revert parallel checkout as the default.
This is because buildbot can't parse gclient output with parallel checkout,
causing got_revision to not be set, causing some scripts to fail.

TBR=bradnelson
TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3325023

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59121 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 3b84d4c9e4 Enable parallel checkout by default to --jobs=8 and smoke tests.
BUG=54084
TEST=updated smoke test and added a new one + stdout untangling

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=59002

Review URL: http://codereview.chromium.org/3357020

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59116 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 5b3f8851ef Automatically disable progress for non tty output, i.e. when it's redirected.
Otherwise, a VT100 terminal will soon be assumed.

TEST=touch foo.txt; gclient sync >> foo.txt &; tail -F foo.txt doesn't show progress.
BUG=none

Review URL: http://codereview.chromium.org/3310026

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@59101 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 3742c84587 Add the infrastructure necessary to support annotated stdout.
Simplify ExecutionQueue.run() by moving code into utility functions.

Reduce the amount of code in WorkerThread.run() to improve reliability.

Don't trap exceptions in single-threaded usage!

BUG=54084
TEST=none

Review URL: http://codereview.chromium.org/3336015

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58974 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org db111f71cf Add automatic auto-flushing stdout.
An automated work around for python lack of stdout flushing.
This design simplifies life once parallel checkout is enabled.

TEST=none
BUG=none

Review URL: http://codereview.chromium.org/3342024

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58819 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 6985efce6e Fix small nits.
pylint is useful to catch errors before they go live so fix little annoyances.

BUG=none
TEST=none

Review URL: http://codereview.chromium.org/3372001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58817 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 158040918c Revert "Do not blindly assume that a .gclient file in a parent directory belongsto the cu..."
This reverts commit 58371

TBR=jochen
BUG=
TEST=

Review URL: http://codereview.chromium.org/3357005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58372 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
jochen@chromium.org 2509f86654 Do not blindly assume that a .gclient file in a parent directory belongs to the current directory.
BUG=54238
TEST=GClientSmoke.testWrongConfig

Review URL: http://codereview.chromium.org/3300007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58371 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org d765d9c848 Revert "Do not blindly assume that a .gclient file in a parent directory belongs to the cur..."
This reverts commit 58352.

TBR=jochen
BUG=breaks build bots

Review URL: http://codereview.chromium.org/3334005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58355 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
jochen@chromium.org 9806750e14 Do not blindly assume that a .gclient file in a parent directory belongs to the current directory.
BUG=54238
TEST=GClientSmoke.testWrongConfig

Review URL: http://codereview.chromium.org/3304007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58352 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 17d01795a6 Cleanup the code in gclient_utils to standardize on CheckCall nomenclature.
Simplify code by removing fail_status
Rename print_messages to always
Simplify the doc.

Review URL: http://codereview.chromium.org/3104036

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@58201 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 2b9aa8e0b3 Refactor SubprocessCallAndFilter() to remove positional arguments.
This way the function is much more similar to subprocess.call(). Further changes
will be done, to be able to convert all the function, to be able to parallelize
output without corrupting it.

Used pylint to verify call graph correctness, causing some other unrelated
changes.

TEST=unit tests
BUG=none

Review URL: http://codereview.chromium.org/3117039

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57369 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 559c3f89b6 Add stdout param to SubprocessCallAndFilter().
It will eventually replace 'print_messages' and be used to annotate printed
lines when using --jobs.

Review URL: http://codereview.chromium.org/3174020

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@57092 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 9e5317ac38 Add --jobs support to gclient. --jobs=1 is still the default for now.
Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.

gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.

--jobs=1 is single threaded as before, albeit with a different code path.

Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.

TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.

---

Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.

           runs with -f        runs with -m      without -f nor -m
 args          real    user      real    user      real    user
 -j 12       20.59s  18.00s     5.64s   7.95s     5.86s   8.10s
        #1 1m05.26s  20.02s     5.20s   7.94s     5.10s   8.09s
             22.79s  18.17s
 -j 1   #2 1m47.00s  16.72s     9.69s   5.72s    12.35s   5.96s
           1m31.28s  17.06s     9.54s   5.85s    10.51s   6.20s
           1m31.79s  16.39s
 before #3 1m30.94s  16.74s     9.77s   5.83s    10.45s   5.77s
           1m30.17s  17.30s    10.36s   5.68s    10.16s   5.88s
 hook #4      8.52s  7.93s
              8.73s  8.13s

#1 This particular run synched to r56023, a webkit roll updating layout tests.
   It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
   manually to compare. Hooks are still run in a single thread. I didn't rest
   'gclient runhooks'.

I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.

Review URL: http://codereview.chromium.org/3135014

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 80cbe8b7d1 Move ExecutionQueue and WorkItem to gclient_utils.py
Review URL: http://codereview.chromium.org/3107009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56020 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org ce46489487 Remove semicolons from revinfo.
I've never known why there were added in the first place

Review URL: http://codereview.chromium.org/3155008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55896 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org f50907b9d1 Fix duplicate Dependency that appeared by switching from a flat list to a tree.
Instead of keeping the reference information in the parent, note for each Dependency if it should be processed with self.should_process. I had to hack a bit with the hooks to also enforce recursion_limit() to keep the old behavior, otherwise From() could cause hooks to run that weren't run before.

BUG=50015
TEST=hooks are run properly. Tested with webkit and pagespeed and buildbot master, fixes running the same checkout multiple times.

Review URL: http://codereview.chromium.org/3124017

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55895 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 049bcedffe Add new class ExecutionQueue to take care of out-of-order execution.
ExecutionQueue takes care of reordering execution depending on each Dependency
requirements. There is no need to sort anymore.

ExecutionQueue also manages the Progress instance and is multithread safe, in preparation to parallel checkout.

Review URL: http://codereview.chromium.org/3112002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55882 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org b1e315f9e8 Add revinfo --actual to fetch the actual revisions,
gclient revinfo alone will now not return the actual checked out revisions but simply what's in the .gclient&DEPS.

BUG=none
TEST=add a --snapshot test to make sure it is not broken

Review URL: http://codereview.chromium.org/3130007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55760 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 6da25d0d81 A small refactor I had lying around for a while.
No functional change.

Review URL: http://codereview.chromium.org/3179001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55745 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 621939b680 Add Dependency.requirements which will solve the issues found in pagespeed and parallel checkout.
Review URL: http://codereview.chromium.org/3159002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55614 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org dde32eecf5 Improve logging and exceptions.
No functional change.

Review URL: http://codereview.chromium.org/3109003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55581 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org f3abb80ae0 Fix the type of 2 exceptions.
Add Dependency.processed and .hooks_ran to ease debugging.

Accept when more than one reference exist for FromImpl(), it may happen as it is the case for pagespeed.

Not much behavior change.

Review URL: http://codereview.chromium.org/3126003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55575 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org da7a1f902a Fix logic error in LateOverride().
It would modify self.parsed_url even when it's called for a From() keyword

Review URL: http://codereview.chromium.org/3171001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55574 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 0d81244c42 Remove default arguments to Dependency.__init__().
Also rename an argument to better fit reality.

No functional change

Review URL: http://codereview.chromium.org/3108001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@55559 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org defec8e689 Disable the duplicate dependency check to get pagespeed back to sync.
BUG=50015
TBR=bradnelson

Review URL: http://codereview.chromium.org/3013031

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53435 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 861fd0feb1 Make file_list recursive so that pattern in hooks will trigger on dependencies too.
This was the behavior before the refactor.

TEST=sync parfait and hooks should be run when necessary.
TBR=bradnelson

Review URL: http://codereview.chromium.org/2819062

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53430 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
bradnelson@google.com 5f8f2a8402 Reverting 53377, horks webkit waterfall.
BUG=None
TEST=None
TBR=maruel

Review URL: http://codereview.chromium.org/3005027

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53419 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 044f4e3998 Fix misalignment
NO CODE CHANGE.

TBR=bradnelson

Review URL: http://codereview.chromium.org/2854061

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53399 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 98d05fa37e Fix a bug with From('..', None) not working.
This issue is exhibited by pagespeed.

Also a solution is always a direct dependency by definition.

TEST=none
TBR=bradnelson

Review URL: http://codereview.chromium.org/3036013

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53397 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org c57e4f2848 Make Dependency.subtree() a standalone function for later use.
No behavior change.

TEST=all test still pass
TBR=bradnelson

Review URL: http://codereview.chromium.org/3005026

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53392 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org bc2d2f908b Fix docstring to be a docstring.
TBR=bradnelson

Review URL: http://codereview.chromium.org/2808062

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53386 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 85c2a19833 Add comments to clarify the purpose of some member variables.
NO CODE CHANGE.

TBR=bradnelson

Review URL: http://codereview.chromium.org/3014027

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53382 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org bffb9042d6 Add Dependency.hierarchy() to better visualize the origin of a dependency.
TBR=bradnelson

Review URL: http://codereview.chromium.org/2883035

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53379 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org cefc9d9113 Don't sort dependencies by name anymore since it can be an issue for some kind of DEPS setup.
PageSpeed is an example of such ordering dependency.

TBR=bradnelson

Review URL: http://codereview.chromium.org/2809072

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53377 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org df2b31582f Reapply gclient.py refactor for the third time.
Bring some OOP and sanity to gclient.py.

- Changed the algorithm from breadth-first to depth-first.
- Added infinite recursion support.
- Fixed From() of From() dependency.
- Fixed cross solution custom deps aliasing.
- Removed support code for old .gclient_entries format.
- Removed IsGitCheckout() in favor for a direct check for .git presence.

TEST=all test pass

The main difference with r51760 is that new smoke tests have been added since which replicates how webkit checkouts with gclient.

Review URL: http://codereview.chromium.org/2917009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@53205 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
mazda@chromium.org 67a1a74d7e Fix gclient.py so that save_pinned_deps works.
Chromium OS continuous builds are failing because of the following error.

Traceback (most recent call last):
  File "/b/depot_tools/gclient.py", line 1212, in <module>
    sys.exit(Main(sys.argv[1:]))
  File "/b/depot_tools/gclient.py", line 1202, in Main
    return command(parser, argv[1:])
  File "/b/depot_tools/gclient.py", line 1122, in CMDrevinfo
    client.PrintRevInfo()
  File "/b/depot_tools/gclient.py", line 776, in PrintRevInfo
    url = entries.pop(name)
KeyError: 'chromiumos'

This change will fix the error.

TEST=none
BUG=none
Review URL: http://codereview.chromium.org/2892007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52180 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org baa578e3d0 Add testing for a solution entry where url = None.
This is being abused by WebKit.

TEST=new smoke test

Review URL: http://codereview.chromium.org/2923005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52096 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org 9b13351cd7 Add testing for the From(File()) case, fix revinfo.
This case covers the webkit gclient usage.

TEST=improved smoke test

Review URL: http://codereview.chromium.org/2968005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52092 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago
maruel@chromium.org dc7445dd70 Revert the refactor (r52005) again, it broke webkit checkout.
TBR=msb

Review URL: http://codereview.chromium.org/2973001

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@52011 0039d316-1c4b-4281-b951-d872f2087c98
15 years ago