Compiling and Running Hadoop MapReduce using Java 8

This is a post summarizing my experience trying to compile the source code for Hadoop using Java8 and running the Hadoop system using Java 8.

I am using an older version of Hadoop, hadoop-1.0.3 and developer preview JDK 8 from Oracle.

The first thing I had to fix is the build.xml file in HADOOP_HOME directory (the base directory). I get the error saying

“class javac1.8”, it turned out to be a bug in ant library. My ant was 1.6.0. I upgraded my ant to 1.9.2 and it is gone. It is a known issue.

 

The next thing I had to fix are the source files. Apparently Java 8 have stricter requirement on generic types. I had to fix two files (Note: if you are using a more recent version of Hadoop, these bugs have been fixed for java8 support). 

src/mapred/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java

src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java

The fixes are pretty simple and you can google the error messages to find the relevant fix in Hadoop bug reports.

 

Once I did all these, I get errors when I actually use 1.8 lambda syntax in my source code modifications. I had to set

<property name=”javac.version” value=”1.6″/> to

<property name=”javac.version” value=”1.8″/> to enable using lambda syntax in the source code I did.

After this, I tried to run Hadoop and I got the rror

“Exception in thread “main” java.lang.UnsupportedClassVersionError: org/apache/hadoop/util/PlatformName : Unsupported major.minor version 52.0″

I googled around and realized that the problem is that I compiled the source code using Java 1.8 but I was running it using 1.6. I changed the JAVA_HOME variable set in conf/hadoop-env.sh to point to the JAVA8 runtime and it works now!

Hope this helps

Cheers!

Yunming

Advertisements
This entry was posted in Hadoop Research, HJ-Hadoop Improvements. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s