The Adventures of a Systems Engineer

I decided to setup a Hadoop cluster and write a MapReduce job  for my distrbuted systems final project. I had done this before with an earlier release and it was fairly straight forward. It turns out it is still straight forward with Hadoop 0.20.2, but the process is not well documented and the configuration has changed. Hopefully I can clear up the process here.

What is Hadoop MapReduce?

MapReduce is a powerful distributed computation technique pioneered by Google. Hadoop MapReduce is an open source implementation written in Java that is maintained by the Apache Software Foundation. Hadoop MapReduce consists of two main parts: the Hadoop distrbuted file system (HDFS) and the MapReduce system.

Getting Hadoop

The first step is to download Hadoop. Go to http://hadoop.apache.org/mapreduce. It is worthwhile to read up on how Hadoop and MapReduce work before you move onto the installation and configuration.

Plan The Installation

Before the actual installation there is a bit of planning to be done. Hadoop works best when run from a local file system. However for convienceince it is also nice to have a common NFS file share to save configuration and log files. Below is an image of what I setup. For the distributed setup at least two nodes are required.

Initial Setup

Before doing any setup of the actual Hadoop system there is some initial setup that needs to be completed, namely the creation of a directory on each node and a shared ssh key. The first step is the easiest. A hadoop install directory needs to be created on each nodes that is going to be a part of the system. The directory must have the same name and location on each node. It is recommended not to use an NFS file share for the installation directory as it can affect performance.

After the install directory has been created a shared ssh key needs to be generated on each node and added to the authorized_hosts file. This allow for passwordless ssh login and is required by the Hadoop cluster startup scripts.

Open Firewall Ports

Hadoop requires a number of ports to be open for the system to work.

Configuration Files

There are three main configuration files that need to be edited: hdfs-site.xml, mapred-site.xml, and core-site.xml. Each file resides in the conf folder where Hadoop is extracted from. There are a lot of parameters that can go into each file but only a few basic ones needs to be set. I have provided my configuration files below. The final file that needs to be edited is hadoop-env.sh, which is a shell script that sets up Hadoop environment variables. At the very least the $JAVA_HOME variable needs to be uncommented and properly set.

core-site.xml

hdfs-site.xml

mapred-site.xml

hadoop-env.sh

Set the Slaves and Master

The master node needs to be defined in the hadoop_dir/conf/masters file. Each slave node needs to be hadoop_dir/conf/slaves file, one machine name/IP address per line.

Deploy the Installation and Configuration Files

The installation and configuration files need to be deployed to each node in the cluster. The easiest way to do this is through scp. I wrote the script below so that I could run a command on each node in my cluster. Another alternative is the Cluster SSH program (cssh). Either approach is preferable to logging onto each node to run  a command.

Using my run_comm.sh script I ran scp on each node in the cluster:

./run_comm.sh "scp -r ~/hadoop /opt/hadoop/hadoop"

This runs the command in quotes on each node in the cluster. In this case I copied the Hadoop installation fom the NFS share (my home directory) to a local directory on each node.

run_comm.sh

Formatting the NameNode

Now that the Hadoop files are on each node the NameNode can be formatted to setup the Hadoop File System.

hadoop_dir/bin/hadoop namenode -format

Starting the Hadoop File System

Now that the namenode has been formatted the distributed file system (DFS) can be started. This is done by using the start-dfs.sh script in the bin directory of the Hadoop installation.

hadoop_dir/bin/start-dfs.sh

The status of the Hadoop File System can be viewed from the administrative page on on the master server, http://master_server:50070.

Starting the MapReduce System

The final step to setting up MapReduce is to start the MapReduce system. This is done by using the start-mapred.sh script that is located in the bin directory of the Hadoop installation.

hadoop_dir/bin/start-mapred.sh

The status of the MapReduce system can be viewed from the administrative page on on the master server, http://master_server:50030.

Submitting a MapReduce Job

Now that the cluster is up and running it is ready to start accepting MapReduce jobs. This is done using the hadoop executable from the bin directory of the Hadoop installation and a jar file that contains a MapReduce program. An example of running the WordCount demo program provided with Hadoop is shown below.

hadoop_dir/bin/hadoop jar jar_location/wordcount.jar org.myorg.WordCount /file_dir_in_hdfs /output_dir_in_hdfs

By default the Xen Hypervisor uses the credit CPU scheduler. The sEDF (simple earliest deadline first) CPU scheduler can be used by setting the sched boot parameter equal to sedf. The easiest way to do this is by editing the grub boot loader config file, menu.lst. The file is located in /boot/grub/menu.lst. The file must be edited by a root user (this can be done in Ubuntu by using the sudo command). This is demonstrated in the entry below. I just copied the existing Xen boot entry and added this one with the sched parameter so I could choose between schedulers at boot time.

title    Xen 3.3 / Ubuntu 8.04.3 LTS, kernel 2.6.24-26-xen sEDF SCHEDULER

root     (hd1,0)

kernel   /boot/xen-3.3.gz sched=sedf

module   /boot/vmlinuz-2.6.24-26-xen root=UUID=e8c13945-5158 console=tty0

module   /boot/initrd.img-2.6.24-26-xen

quiet

I have been setting up a test Xen 3.3 box running on Ubuntu 8.04. I have been getting the following error: “Device xxxx (vbd) cannot be connected. failed to find an unused loop device”  where xxxx is some number. It turns out that by default only 8 loopback devices are enabled. The following command shows how many loopback devices are enabled:

ls -l /dev/loop*

To increase the number of loopback devices the following line needs to be added to the /etc/modules file. I used a value of 64 so I can run up to 32 virtual machines.

add loop max_loop=64

Now when you run ls -l /dev/loop* you will see additional loopback devices.

This is a simple tip, but definitely worth mentioning. Typing shutdown -r now in a terminal will restart a Linux system.