Reading JVM Output in Hadoop

This post works with Hadoop 1.0.3

I was trying to read the output from garbage collection verbose (-verbose:gc) on the ChildJVMs (the JVMs in Hadoop that actually execute the map/reduce tasks).

There is a log directory that you could set in conf files. The directory I set was /tmp/logs

To find the output from JVM, go to

logs/userlogs/job_date_time_jobid/attempt_xx  (Each job corresponds to one start of the Hadoop system, hdfs, tasktracker, jobtracker and all that, each attempt corresponds to one job submission for the system).

There will be a few files in the directory

log.index  stderr  stdout  syslog

The information we need is in stdout, following is an example

[GC 3072K->480K(8896K), 0.0020710 secs]
[GC 3552K->528K(11968K), 0.0019410 secs]
[GC 6672K->968K(11968K), 0.0021060 secs]
[GC 7112K->1304K(18112K), 0.0020390 secs]
[GC 13592K->1544K(18112K), 0.0068730 secs]
[GC 13832K->2216K(30720K), 0.0055210 secs]

This entry was posted in Hadoop Research, HJ-Hadoop Improvements, Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s