Using external libraries in Hadoop

This is a post summarizing my attempt to use a third party library in modifying the source code for Hadoop.

I was trying to reimplement the changes I made to Hadoop using Habanero Java library, a parallel programming library developed at Rice university by Shams Imam.

In order to use the library, I looked through the build.xml file (with the help of Shams) and noticed the following entries


<!– the normal classpath –>

<path id=”classpath”>

<pathelement location=”${build.classes}”/>

<path refid=”src.lib.classpath”/>

<pathelement location=”${conf.dir}”/>



<path id=”src.lib.classpath”>

<fileset dir=”${lib.dir}”>

<include name=”**/*.jar” />

<exclude name=”**/excluded/” />


<path refid=”ivy-common.classpath”/>



The most relevant part is the second entry, src.lib.classpath. It indicates that it is using all of the .jar files in the directory.  It indicates that we could just put the .jar file (habanero-java-library.jar) in the class path directory and Hadoop will automatically pick it up.

Notice the file set property points to a “lib.dir” directory.

As we try to track down where the lib.dir is , we find the following entry

<property name=”lib.dir” value=”${basedir}/lib”/>

This indicates the directory where we should put our jar files in in HADOOP_HOME/lib. I put the jar in there, and future ant commands works fine to compile the Hadoop source code.


A comment from Shams about Ant

” ant does classpaths 2 ways,  either manually adding entry or using filesets (as this build.xml is doing). ” 

In the above post, we are doing the first approach.


This entry was posted in Hadoop Research, HJ-Hadoop Improvements. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s