I have been working with the Eclipse build a lot lately. It takes about 18 minutes to build eclipse on my laptop so every time I made a change to the rpm or the build scripts I had to wait 18 minutes to see the results.
So I wanted to speed the build up. I was under the impression that the eclipse build was IO bound so to speed that up I decided to use . I created a virtual machine with tons of memory and allocated a 4GB folder:
mkdir /tmp/build mount -t ramfs -o size=8g ramfs /tmp/build
There was no speed up. And I had made sure that there was no swapping.
So after digging around a little bit I found a way to analyse the build better. At first I had wanted to use
iostat
but I discovered dstat.
dstat has an output which is easier to parse for graphic tools:
$ dstat -tc
----system---- ----total-cpu-usage----
time |usr sys idl wai hiq siq
29-11 09:34:12| 4 2 92 1 0 0
29-11 09:34:13| 4 2 91 2 1 0
The each line in the output tells it what percentage of the time since the last sample your cpu spent running user process, system process, idling, waiting for io respectively. I don’t really know what the last two values are 
.
So I ran:
dstat -tc >& dstat.raw
In retrospect I really did not need to sample ever second and should have run
dstat -tc 5 >& dstat.raw
to sample every 5 seconds.
So I imported dstat.raw into gnumeric and ta-da!:
The important colours above graph are the yellow (idling cpu) the blue (cpu executing eclipse build) and the turquoise (cpu waiting for io). The first thing you will notice is that there is not a lot of io blockage. Of note are the big peak in the beginning which I assume is the extraction of the sources tar ball and at the end which I assume is the writing of the rpms. The second thing you’ll notice is that there is a lot of high yellow which is evidence to the fact that my cpu’s spend most of the time idling during the build. Finally you’ll note that the blue never goes above 25% and is usually around 20%. This is because I have 4 cpu’s and the eclipse build is not parallelized.
So in conclusion eclipse build is cpu bound, and the most obvious solution to this is to make the build parallel where possible.
November 29, 2011 at 7:53 pm |
Hi Sami
I’m the release engineer for the Eclipse project. Interesting graphs
What are the specs on your laptop?
There is parallelization in the build. We execute many Ant calls in parallel which you can see in the main target of org.eclipse.releng.eclipsebuilder/buildAll.xml. I’m sure the scripts you use to create the RPMs are different. In addition, we use
flattenDependencies=true
parallelCompilation=true
when compiling our bundles to speed up the process.
November 29, 2011 at 9:47 pm |
Hi Kim
Hmm… Let me look into why we are not taking advantage of the parallelization in the scripts. Maybe these flags are what we are missing.
The scripts we use are wrappers around the eclipse scripts.
My laptop is core i7. 4 cpu’s @ 2.67GHz and 8GB ram
November 30, 2011 at 10:01 am |
FYI, when we performed lots of parallel builds + test for Bonita Studio (RCP application), we identified that CPU was a limitation, but with a 12-cores machine, it was not the bottleneck to run builds in parallel.
The identified bottleneck for our use-case was IO access. Then we could save about 30% by moving to SSD disks and also, and mainly, by having different disks for builds. So that if several builds are performed at the same time, on a 12-cores machine and with different disks, there is not a lot of shared resource to wait for, and everything goes much faster.
November 30, 2011 at 2:25 pm |
Nice, I just have to figure out how to make our build parallel, then look at the next bottleneck. How long does an eclipse build take on your setup ?
December 1, 2011 at 12:52 pm |
You can test it by just putting the 2 properties Kim mentioned into build.properties. I’m pretty sure that I had this when I initially worked on eclipse-build (3.5 timeframe).
November 30, 2011 at 3:02 pm |
Hi,
I’m still working @Bonitasoft . Mickaël is a former colleague highly implied in the build of our product.
So here is some more information about the build:
- I don’t have the full details of all phases time for the build
- we have two product BOS and BOS-SP available for linux, windows and mac and zip for all platforms
- we are generating code based on EMF and GMF models during build phase
- When running tests, we don’t provide all platforms.
- Several builds can be run in paralell
And here are time:
- BOS all-in-one: 20min to 30min
- BOS-SP all-in-one: 40min to 1hour
- BOS with tests: 37min to 50min
- BOS-SP with tests: 45min to 1 hour
Some ideas to go further is to use maven tycho build in order to not have to rebuild some common plugins between BOS and BOS-SP.(we tried with p2 but we can’t have a zip version compatible with all platforms)
Hope it helps
Regards,
December 4, 2011 at 10:03 pm |
The blog post I wrote when we first introduced the parallel compilation can be found here:
http://aniefer.blogspot.com/2008/10/sorting-bundles-and-parallel.html
December 6, 2011 at 2:53 pm |
Thanks Andrew! I’ll take a look when I am debugging the linux eclipse build.
December 6, 2011 at 8:07 am |
I’ve been looking for years for a way to build Eclipse.
Where can I find the source for this build? Are there instructions how to repeat the process on my machine?
December 6, 2011 at 2:57 pm |
Hi Aaron,
Are you on a Linux system ? The code can be found here: http://wiki.eclipse.org/Linux_Tools_Project/Eclipse_Build
That page will tell you how to get the source and how to build.
You can find a bit of documentation here: http://wiki.eclipse.org/Linux_Tools_Project/Eclipse_Build/Inner_Workings
December 6, 2011 at 5:12 pm |
Yes, I’m on Ubuntu 11.10 (oneiric)
I followed the instructions and got stuck at:
org.eclipse.linuxtools.eclipse-build/eclipse-build/build.xml:369: Could not find suitable system JAR for org.apache.el_7.0.21.jar. Tried: /usr/share/java/tomcat-jasper-el.jar
The file exists (I had to replace a “/” with “-”, though). Any ideas?
December 6, 2011 at 6:47 pm |
Yes, you have to append that file to the search path in dependencies.properties find the file which corresponds to the missing dependency (org.apache.el_7.0.21.jar) and put the path to the file provided by your distro at the end of that line separated by a colon. You should join #eclipse-linux on freenode if you have more questions
January 19, 2012 at 9:34 pm
I had this same error, and even though I added the path (after a colon), I’m still getting the error:
Could not find suitable system JAR for org.apache.el_7.0.21.jar. Tried: /usr/share/java/tomcat/jasper-el.jar, /home/cory/apache-tomcat-6.0.35-src/output/build/lib/jasper-el.jar
By the way, I couldn’t even find jasper-el.jar on my system so I downloaded the src for apache tomcat (I googled to find where jasper-el.jar came from)
January 19, 2012 at 9:47 pm |
Cory,
On Fedora jasper-el.jar is provided by tomcat-lib-7.0.23-1.fc16.noarch. Otherwise the way to debug this is to look at the manifest.mf found in eclipse-build/dependencyManifests/org.apache.el_7.0.21.jar/ and compare that with the manifest from the system jar you are proposing. Compare the bundle names and the bundle versions, and you will find why eclipse-build is refusing to use that file. A good place to download jars from is http://www.eclipse.org/orbit/
Aaron,
Have you had any luck building eclipse using eclipse-build ?
January 21, 2012 at 11:56 am |
No, I have the same problem (no Orbit MANIFEST in Ubuntu’s JARs).
Why do you insist on distribution-specific files for a build when the final product isn’t going to use these files anyway?
January 21, 2012 at 2:34 pm
Eclipse will use the jars it links to during the build. If there is a file which we require for the build which is not used by Eclipse at run time, then the requirement is probably there for historical reasons and should be removed. Let me know of such cases and I will look into it.
January 19, 2012 at 11:32 pm |
Sami,
Thanks for helping me. I’m using Ubuntu 11.10. I am not very familiar with manifest files (I found a great tutorial on the Oracle website though).
My jasper-er.jar’s manifest file doesn’t mention any bundle name or version headers. Is causing the problem?
January 20, 2012 at 3:36 pm |
I usually compare Bundle-SymbolicName and Bundle-Version. If the manifest file is empty then that is the issue. Try downloading the bundle from Orbit and/or file a bug with the package owner to fix the manifest files