Hadoop - shawfdong/hyades GitHub Wiki
The goal is to set up a distributed multi-node Apache Hadoop cluster[1] on the 8 GPU nodes, gpu-1 – gpu-8, in Hyades. Each GPU node is equipped with two (2x) 8-core Intel Sandy Bridge Xeon E5-2650 processors at 2.0 GHz, 64 GB memory, two (2x) 500GB hard drives; and one Nvidia K20 GPU. On each GPU node, an ext4 file system is created on the second hard drive and is mounted at /data. Those spaces will be used exclusively by Hadoop.
We'll install Apache Hadoop 2 or YARN, which is the new version of the open-source software framework for distributed storage (HDFS) and distributed processing (MapReduce) of Big Data on clusters of commodity hardware.
- Masters
- NameNode: gpu-1, which manages the Hadoop Distributed File System (HDFS) namespace and regulates access to files by clients.
- ResourceManager: gpu-1, which arbitrates resources among all the applications in the cluster.
- Secondary NameNode: gpu-2, which performs periodic checkpoints of the namespace and helps keep the size of file containing log of HDFS modifications within certain limits at the NameNode.
- Slaves
- DataNode: gpu-1 – gpu-8, which are the slaves of the NameNode, and manage storage attached to the nodes that they run on.
- NodeManager: gpu-1 – gpu-8, which are the slaves of the ResourceManager. The ResourceManager (RM) and per-node slave, the NodeManager (NM), form the data-computation framework.
Java is already present on all nodes in Hyades:
$ echo $JAVA_HOME /usr/java/latest $ java -version java version "1.7.0_13" Java(TM) SE Runtime Environment (build 1.7.0_13-b20) Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
We will use a dedicated Hadoop user account (hduser) for running Hadoop:
$ id hduser uid=510(hduser) gid=510(hadoop) groups=510(hadoop)
We use host-based ssh key authentication[2] on Hyades, which allows for password-less SSH access between nodes. There is no need to create individual SSH keys for each user account.
Download Hadoop 2.5.2 from ones of the mirrors:
$ cd /scratch/ $ wget http://apache.claz.org/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
Unpack the tar ball onto the Lustre file system (/pfs), which is mounted on all nodes in Hyades:
$ mkdir -p /pfs/sw/bigdata $ tar xvfz /scratch/hadoop-2.5.2.tar.gz -C /pfs/sw/bigdata
Fix native libraries:
In order to improve performance, Hadoop tries to load native implementations of certain components[3]. These components are available in dynamically-linked native libraries, located in the lib/native directory. Although the native libraries provided by the official Hadoop 2.5.2 release are 64-bit, they are linked with GLIBC_2.14. The glibc on RHEL/CentOS 6, however, is version 2.12. Thus the stock native libraries can't be loaded and we'll get the following warning:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableThe fix is described in Building Hadoop from source.
Change the ownership of the Hadoop installation:
# chown -R hduser:hadoop /pfs/sw/bigdata/hadoop-2.5.2/
Create a module file (/pfs/sw/modulefiles/hadoop/2.5.2) that sets the following environment variables:
HADOOP_HOME=/pfs/sw/bigdata/hadoop-2.5.2 HADOOP_VERSION=2.5.2 PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH HADOOP_LOG_DIR=/data/logs YARN_LOG_DIR=/data/logsNOTE by default the logs are saved at $HADOOP_HOME/logs, which could swamp the Lustre file system; we define the last two environment variables so that logs will be saved locally on each node.
Append the following line to ~hduser/.bashrc:
module load hadoop
The file lists the hosts, one per line, where the Hadoop slave daemons (DataNode & NodeManager) will be run:
gpu-1 gpu-2 gpu-3 gpu-4 gpu-5 gpu-6 gpu-7 gpu-8
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://gpu-1:8020</value>
<description>The URI of the default file system</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>The size of buffer for use in sequence files</description>
</property>
</configuration>
NOTE
- We designate gpu-1 as the NameNode.
- The default values are defined in $HADOOP_HOME/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>gpu-1</value>
<description>The hostname of the ResourceManager</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service for MapReduce</description>
</property>
</configuration>
NOTE
- We designate gpu-1 as the YARN ResourceManage.
- We set MapReduce shuffle as the auxiliary service[4].
- The default values are defined in $HADOOP_HOME/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml.
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/nn</value>
<description>
Determines where on the local filesystem the DFS name node should store the name table (fsimage)
</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data/dn</value>
<description>
Determines where on the local filesystem a DFS data node should store its blocks
</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>gpu-2:50090</value>
<description>
The secondary namenode http server address and port (default is 0.0.0.0:50090)
</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>Default block replication</description>
</property>
<property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
</configuration>
NOTE
- We designate gpu-2 as the Secondary NameNode.
- The default for dfs.permissions.superusergroup is supergroup and by default, the group of all files and directories in HDFS is supergroup; but that group does not exist on Hyades cluster. We set its value to hadoop.
- The default values are defined in $HADOOP_HOME/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs</description>
</property>
</configuration>
The default values are defined in $HADOOP_HOME/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml.
Format the HDFS:
[root@hyades ~]# ssh gpu-1 [root@gpu-1 ~]# su - hduser [hduser@gpu-1 ~]$ hdfs namenode -formatThis only needs to be done once.
To start HDFS daemons, run on gpu-1:
[hduser@gpu-1 ~]$ start-dfs.shwhich will start the NameNode daemon on gpu-1, DataNode on all the slaves, and Secondary NameNode on gpu-2.
By default, the HDFS is only writable by hduser:
[hduser@gpu-1 ~]$ hdfs dfs -ls -d / drwxr-xr-x - hduser hadoop 0 2014-12-06 21:49 /
Make the HDFS world writable with sticky bit:
[hduser@gpu-1 ~]$ hdfs dfs -chmod 1777 /
To start all YARN daemons, run on gpu-1:
[hduser@gpu-1 ~]$ start-yarn.shwhich will start the ResourceManager daemon on gpu-1, NodeManager on all the slaves.
To start the MapReduce JobHistory Server, run on gpu-1:
[hduser@gpu-1 ~]$ mr-jobhistory-daemon.sh start historyserver
To stop HDFS daemons, run on gpu-1:
[hduser@gpu-1 ~]$ stop-dfs.sh
To stop all YARN daemons, run on gpu-1:
[hduser@gpu-1 ~]$ stop-yarn.sh
To stop the MapReduce JobHistory Server, run on gpu-1:
[hduser@gpu-1 ~]$ mr-jobhistory-daemon.sh stop historyserver
Apache Hadoop provides an HDFS NFS Gateway that supports NFSv3 and allows HDFS to be mounted as part of the client's local file system[5]. We'll run the HDFS NFS Gateway on gpu-1, and mount the exported file system on the master node Hyades.
Add the following to $HADOOP_HOME/etc/hadoop/core-site.xml:
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>*</value>
<description>The 'hduser' user is allowed to proxy any group.</description>
</property>
<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>gpu-1</value>
<description>This is the host where the nfs gateway is running.</description>
</property>
Add the following to $HADOOP_HOME/etc/hadoop/hdfs-site.xml:
<property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
<property>
<name>nfs.exports.allowed.hosts</name>
<value>10.6.0.0/16 rw</value>
</property>
NOTE using 10.6.0.0/16 rw,no_root_squash as value for nfs.exports.allowed.hosts doesn't seems to have the desired effect and root is still squashed nonetheless!
Restart HDFS daemons:
[hduser@gpu-1 ~]$ stop-dfs.sh [hduser@gpu-1 ~]$ start-dfs.sh
Stop the system rpcbind service on gpu-1 (as root):
[root@gpu-1 ~]# service rpcbind stop
Start portmap provided by Hadoop on gpu-1 (as root):
[root@gpu-1 ~]# hadoop-daemon.sh --script hdfs start portmapNOTE hadoop-daemon.sh start portmap and hadoop portmap are deprecated. Run either hadoop-daemon.sh --script hdfs start portmap or hdfs portmap.
Start nfsd & mountd provided by Hadoop on gpu-1 (as hduser):
[hduser@gpu-1 ~]$ hadoop-daemon.sh --script hdfs start nfs3NOTE hadoop-daemon.sh start nfs3 and hadoop nfs3 are deprecated. Run either hadoop-daemon.sh --script hdfs start nfs3 or hdfs nfs3.
Mount the exported HDFS on Hyades (as root):
[root@hyades ~]# mkdir /hdfs [root@hyades ~]# mount -t nfs -o vers=3,proto=tcp,nolock,noatime gpu-1:/ /hdfs