Reimplementing HJ-Mapper using pure Java concurrent library

Since Habanero Java language had various compatibility issues with running within a Java thread, I reimplemented the HJ-Mapper using pure Java thread, executor service, static, final variables. Here is  a summary of what I did, the challenges I met,

In the original mapper, I used the basic constructs, such as async, finish. What I observed later is that to replicate the simple semantics of async and finish is not easy in Java, highlighting the importance of providing simple parallel constructs. Here is a series of notes

1. Variables used in the threads have to be declared as final. In HJ, it doesn’t have to be final, a copy will be made automatically. In order to get a local copy of an array, I had to write a block of code to create a copy manually. A side note, to make a shallow copy of an array, it appears that a copy constructor would work.

copyArray = new Array(oldArray);

2. I had to use thread executor service to set a limit on the number of threads to be created. The executor service can use a fixed thread pool to set the number of threads it uses.

3. To simulate a finish scope, again I had to resort to the executor service shut down, await finish api, shown in this post

http://stackoverflow.com/questions/1250643/how-to-wait-for-all-threads-to-finish-using-executorservice

</p><pre><code>ExecutorService taskExecutor = Executors.newFixedThreadPool(4);
while(...) {
  taskExecutor.execute(new MyTask());
}
taskExecutor.shutdown();
try {
  taskExecutor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
  ...
}<br /><br />
Advertisements
This entry was posted in Habanero Java, Hadoop Research, HJ-Hadoop Improvements, Java. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s