Analysing Eclipse Build

I have been working with the Eclipse build a lot lately. It takes about 18 minutes to build eclipse on my laptop so every time I made a change to the rpm or the build scripts I had to wait 18 minutes to see the results.

So I wanted to speed the build up. I was under the impression that the eclipse build was IO bound so to speed that up I decided to use . I created a virtual machine with tons of memory and allocated a 4GB folder:

mkdir /tmp/build
mount -t ramfs -o size=8g ramfs /tmp/build

There was no speed up. And I had made sure that there was no swapping.

So after digging around a little bit I found a way to analyse the build better. At first I had wanted to use

iostat

but I discovered dstat.

dstat has an output which is easier to parse for graphic tools:

$ dstat -tc
----system---- ----total-cpu-usage----
     time     |usr sys idl wai hiq siq
29-11 09:34:12|  4   2  92   1   0   0
29-11 09:34:13|  4   2  91   2   1   0

The each line in the output tells it what percentage of the time since the last sample your cpu spent running user process, system process, idling, waiting for io respectively. I don’t really know what the last two values are :).

So I ran:

dstat -tc >& dstat.raw

In retrospect I really did not need to sample ever second and should have run

dstat -tc 5 >& dstat.raw

to sample every 5 seconds.
So I imported dstat.raw into gnumeric and ta-da!:

The important colours above graph are the yellow (idling cpu) the blue (cpu executing eclipse build) and the turquoise (cpu waiting for io). The first thing you will notice is that there is not a lot of io blockage. Of note are the big peak in the beginning which I assume is the extraction of the sources tar ball and at the end which I assume is the writing of the rpms. The second thing you’ll notice is that there is a lot of high yellow which is evidence to the fact that my cpu’s spend most of the time idling during the build. Finally you’ll note that the blue never goes above 25% and is usually around 20%. This is because I have 4 cpu’s and the eclipse build is not parallelized.

So in conclusion eclipse build is cpu bound, and the most obvious solution to this is to make the build parallel where possible.

About these ads

23 Responses to “Analysing Eclipse Build”

  1. Kim Moir Says:

    Hi Sami

    I’m the release engineer for the Eclipse project. Interesting graphs :-) What are the specs on your laptop?

    There is parallelization in the build. We execute many Ant calls in parallel which you can see in the main target of org.eclipse.releng.eclipsebuilder/buildAll.xml. I’m sure the scripts you use to create the RPMs are different. In addition, we use

    flattenDependencies=true
    parallelCompilation=true

    when compiling our bundles to speed up the process.

  2. Sami Wagiaalla Says:

    Hi Kim :)

    Hmm… Let me look into why we are not taking advantage of the parallelization in the scripts. Maybe these flags are what we are missing.

    The scripts we use are wrappers around the eclipse scripts.

    My laptop is core i7. 4 cpu’s @ 2.67GHz and 8GB ram

  3. Mickael Istria Says:

    FYI, when we performed lots of parallel builds + test for Bonita Studio (RCP application), we identified that CPU was a limitation, but with a 12-cores machine, it was not the bottleneck to run builds in parallel.
    The identified bottleneck for our use-case was IO access. Then we could save about 30% by moving to SSD disks and also, and mainly, by having different disks for builds. So that if several builds are performed at the same time, on a 12-cores machine and with different disks, there is not a lot of shared resource to wait for, and everything goes much faster.

  4. Sami Wagiaalla Says:

    Nice, I just have to figure out how to make our build parallel, then look at the next bottleneck. How long does an eclipse build take on your setup ?

    • Alexander Kurtakov Says:

      You can test it by just putting the 2 properties Kim mentioned into build.properties. I’m pretty sure that I had this when I initially worked on eclipse-build (3.5 timeframe).

  5. scratsh Says:

    Hi,

    I’m still working @Bonitasoft . Mickaël is a former colleague highly implied in the build of our product.

    So here is some more information about the build:
    - I don’t have the full details of all phases time for the build
    - we have two product BOS and BOS-SP available for linux, windows and mac and zip for all platforms
    - we are generating code based on EMF and GMF models during build phase
    - When running tests, we don’t provide all platforms.
    - Several builds can be run in paralell

    And here are time:
    - BOS all-in-one: 20min to 30min
    - BOS-SP all-in-one: 40min to 1hour
    - BOS with tests: 37min to 50min
    - BOS-SP with tests: 45min to 1 hour

    Some ideas to go further is to use maven tycho build in order to not have to rebuild some common plugins between BOS and BOS-SP.(we tried with p2 but we can’t have a zip version compatible with all platforms)

    Hope it helps

    Regards,

  6. Andrew Niefer Says:

    The blog post I wrote when we first introduced the parallel compilation can be found here:

    http://aniefer.blogspot.com/2008/10/sorting-bundles-and-parallel.html

  7. Aaron Digulla Says:

    I’ve been looking for years for a way to build Eclipse.

    Where can I find the source for this build? Are there instructions how to repeat the process on my machine?

  8. Aaron Digulla Says:

    Yes, I’m on Ubuntu 11.10 (oneiric)

    I followed the instructions and got stuck at:

    org.eclipse.linuxtools.eclipse-build/eclipse-build/build.xml:369: Could not find suitable system JAR for org.apache.el_7.0.21.jar. Tried: /usr/share/java/tomcat-jasper-el.jar

    The file exists (I had to replace a “/” with “-”, though). Any ideas?

    • Sami Wagiaalla Says:

      Yes, you have to append that file to the search path in dependencies.properties find the file which corresponds to the missing dependency (org.apache.el_7.0.21.jar) and put the path to the file provided by your distro at the end of that line separated by a colon. You should join #eclipse-linux on freenode if you have more questions

      • Cory Jones Says:

        I had this same error, and even though I added the path (after a colon), I’m still getting the error:

        Could not find suitable system JAR for org.apache.el_7.0.21.jar. Tried: /usr/share/java/tomcat/jasper-el.jar, /home/cory/apache-tomcat-6.0.35-src/output/build/lib/jasper-el.jar

        By the way, I couldn’t even find jasper-el.jar on my system so I downloaded the src for apache tomcat (I googled to find where jasper-el.jar came from)

  9. Sami Wagiaalla Says:

    Cory,

    On Fedora jasper-el.jar is provided by tomcat-lib-7.0.23-1.fc16.noarch. Otherwise the way to debug this is to look at the manifest.mf found in eclipse-build/dependencyManifests/org.apache.el_7.0.21.jar/ and compare that with the manifest from the system jar you are proposing. Compare the bundle names and the bundle versions, and you will find why eclipse-build is refusing to use that file. A good place to download jars from is http://www.eclipse.org/orbit/

    Aaron,

    Have you had any luck building eclipse using eclipse-build ?

    • digulla Says:

      No, I have the same problem (no Orbit MANIFEST in Ubuntu’s JARs).

      Why do you insist on distribution-specific files for a build when the final product isn’t going to use these files anyway?

      • Sami Wagiaalla Says:

        Eclipse will use the jars it links to during the build. If there is a file which we require for the build which is not used by Eclipse at run time, then the requirement is probably there for historical reasons and should be removed. Let me know of such cases and I will look into it.

  10. Cory Jones Says:

    Sami,

    Thanks for helping me. I’m using Ubuntu 11.10. I am not very familiar with manifest files (I found a great tutorial on the Oracle website though).

    My jasper-er.jar’s manifest file doesn’t mention any bundle name or version headers. Is causing the problem?

    • Sami Wagiaalla Says:

      I usually compare Bundle-SymbolicName and Bundle-Version. If the manifest file is empty then that is the issue. Try downloading the bundle from Orbit and/or file a bug with the package owner to fix the manifest files

  11. Barrett Says:

    This post is very informative but it took me a long time to find it in google.

    I found it on 11 spot, you should focus on quality backlinks building,
    it will help you to increase traffic. And i know how to help you, just
    search in google – k2 seo tips and tricks

  12. find related keywords Says:

    I’ve noticed that credit restoration activity needs to
    be conducted with techniques. If not, you will probably
    find yourself endangering your standing. In order to succeed in fixing your
    credit ranking you have to always make sure that from this time you pay your entire monthly costs
    promptly in advance of their planned date.
    It is significant given that by not accomplishing that
    area, all other actions that you will decide to use to improve your credit
    position will not be useful. Thanks for giving your concepts.

  13. Mike Says:

    I read a lot of interesting articles here. Probably you spend a lot of time writing, i know how to save you
    a lot of time, there is an online tool that
    creates high quality, google friendly posts in minutes, just
    type in google – laranitas free content source

  14. obat herbal Says:

    I think this is one of the most significant information for me.

    And i’m glad reading your article. But wanna remark on few
    general things, The site style is ideal,
    the articles is really great : D. Good job, cheers

  15. firma za granicą Says:

    I have been examinating out some of your articles and i must say pretty nice stuff.
    I will surely bookmark your blog.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: