Monthly Archives: November 2013

Research into the implementation of Child JVM

This is  a post detailing the work I did to understand the designs of the child JVM and make possible plans of implementing a “compute server”. What is a “compute server”? It is one server that accepts Tasks from TaskTracker … Continue reading

Posted in Hadoop Research, HJ-Hadoop Improvements, Uncategorized | Leave a comment

COMP 425 MidTerm Review Notes 11 – 23

Lecture 11 -12 More Complex Pipelines A software could resolve data conflict by inserting a NOP. (DELAY) We could still have a structural hazards when two instructions try to write to the data register at the same time. It can … Continue reading

Posted in Architecture, Class | Leave a comment

COMP 425 MidTerm Review Notes 1 – 10

The following is my review notes for the first mid term for Comp 425 Including Lecture 1 to 10. First two lectures are introduction to early computers. Not much notes are taken. Lecture 3 (Single Cycle Professor) There is a … Continue reading

Posted in Architecture, Class | Leave a comment

How does Hadoop create job splits and assign them to tasks?

Coming back to the problem we are trying to solve here, trying to pass multiple InputSplits to one Map Task. It appears that the Task object stayed relatively intact after it is retrieved by the TaskTracker and assigned to the … Continue reading

Posted in Hadoop Research, HJ-Hadoop Improvements, Uncategorized | Leave a comment

How does input split get passed from Task tracker to Child JVM task execution

In the last post, we traced the process a task is assigned to a Tasktracker -> launch new JVM (TaskRunner.java, TaskTracker.TaskController)-> execute the assigned task in the ChildJVM. (Child.java) Now, we delve into how the map and reduce task get … Continue reading

Posted in Hadoop Research | Leave a comment

Hadoop Multiple InputSplit to the same Map Task Research Notes

This post is a summary to my research into the source code of Hadoop. I was investigating what would be the easiest way to parallelize I/O, deserialization and computation in the same Map Task.

Posted in Hadoop Research, HJ-Hadoop Improvements | 1 Comment

Hadoop Heartbeat, Task Tracker, Task Runner exploration notes

HeartBeat Mechanism Task Assignment (pg 192, Hadoop The Definitive Guide, 3rd Edition) Tasktrackers run a simple loop that periodically sends heartbeat method calls to the jobtracker. It is a channel for messages that indicates whether a task tracker is alive … Continue reading

Posted in Hadoop Research, HJ-Hadoop Improvements, Uncategorized | Leave a comment