Skip to content

C
Chromium_Depot_tools
Switch branch/tag
  1. 02 Sep, 2010 6 commits
  3. 01 Sep, 2010 4 commits
  5. 31 Aug, 2010 1 commit
  7. 26 Aug, 2010 2 commits
  9. 25 Aug, 2010 2 commits
  11. 24 Aug, 2010 1 commit
  13. 23 Aug, 2010 3 commits
  15. 22 Aug, 2010 1 commit
  17. 20 Aug, 2010 3 commits
  19. 19 Aug, 2010 4 commits
  21. 18 Aug, 2010 3 commits
  23. 17 Aug, 2010 3 commits
  25. 16 Aug, 2010 4 commits
  27. 13 Aug, 2010 2 commits
    • maruel@chromium.org's avatar
      Add --jobs support to gclient. --jobs=1 is still the default for now. · 9e5317ac
      maruel@chromium.org authored 
      Huge thanks to piman@ for working on a patch. I chose a different design but he
gave me motivation and ideas. Sorry for not accepting his patch earlier, this
was mostly due to broken gclient implementation itself.

gclient can now run an unlimited number of parallel checkouts and always keep
the checkout coherency correct.

--jobs=1 is single threaded as before, albeit with a different code path.

Issues:
- Using --jobs with a value other than 1 will result in a mangled output.
- Exceptions thrown in a thread will be have the wrong stack trace.

TEST=gclient sync -j 99 in a ssh:// chromiumos checkout is dramatically faster.

---

Here's the perf on linux on i7-860 for a chromium checkout with warm cache. Cold
cache will result is significantly reduced improvements so this is best case
improvements. The sync was no-op all the time except where noted. All execution
where with "time gclient sync " + args. Didn't include 'sys' column since it was
statistically insignifiant and highly correlated with 'user'.

           runs with -f        runs with -m      without -f nor -m
 args          real    user      real    user      real    user
 -j 12       20.59s  18.00s     5.64s   7.95s     5.86s   8.10s
        #1 1m05.26s  20.02s     5.20s   7.94s     5.10s   8.09s
             22.79s  18.17s
 -j 1   #2 1m47.00s  16.72s     9.69s   5.72s    12.35s   5.96s
           1m31.28s  17.06s     9.54s   5.85s    10.51s   6.20s
           1m31.79s  16.39s
 before #3 1m30.94s  16.74s     9.77s   5.83s    10.45s   5.77s
           1m30.17s  17.30s    10.36s   5.68s    10.16s   5.88s
 hook #4      8.52s  7.93s
              8.73s  8.13s

#1 This particular run synched to r56023, a webkit roll updating layout tests.
   It's still faster than a no-op sync without parallel checkout.
#2 Maybe there was a sync or computer hickup, I didn't realize.
#3 This is depot_tools@56020
#4 Since -f implies runhooks, I ran the hook 'python src/build/gyp_chromium'
   manually to compare. Hooks are still run in a single thread. I didn't rest
   'gclient runhooks'.

I tried to go a ssh:// checkout of chromium os tree but it timed out everytime I
tried to sync so I couldn't get data points. I expect an order of magnitude of
improvement or more.

Review URL: http://codereview.chromium.org/3135014

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56079 0039d316-1c4b-4281-b951-d872f2087c98
      9e5317ac
    • maruel@chromium.org's avatar
      Move ExecutionQueue and WorkItem to gclient_utils.py · 80cbe8b7
      maruel@chromium.org authored 
      Review URL: http://codereview.chromium.org/3107009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/depot_tools@56020 0039d316-1c4b-4281-b951-d872f2087c98
      80cbe8b7
  29. 12 Aug, 2010 1 commit