Test Cluster and Run MapReduce

In this page, we will try to run the cluster and put files using HDFS and run MapReduce using the Yarn provider.

1. Format HDFS and first run

@ on ALL nodes

hadoop namenode -format

Start HDFS server

@ on MASTER

start-dfs.sh

You should get an output like:

ubuntu@master:~$ start-dfs.sh 
    Starting namenodes on [master]
    master: starting namenode, logging to /home/ubuntu/hadoop/logs/hadoop-ubuntu-namenode-master.out
    slave1: starting datanode, logging to /home/ubuntu/hadoop/logs/hadoop-ubuntu-datanode-slave1.out
    slave2: starting datanode, logging to /home/ubuntu/hadoop/logs/hadoop-ubuntu-datanode-slave2.out
    Starting secondary namenodes [secondarymaster]
    secondarymaster: starting secondarynamenode, logging to /home/ubuntu/hadoop/logs/hadoop-ubuntu-secondarynamenode-secondarymaster.out

Start YARN server

@ on MASTER

start-yarn.sh

You should again get an output like:

 ubuntu@master:~$ start-yarn.sh
    starting yarn daemons
    starting resourcemanager, logging to /home/ubuntu/hadoop/logs/yarn-ubuntu-resourcemanager-master.out
    slave1: starting nodemanager, logging to /home/ubuntu/hadoop/logs/yarn-ubuntu-nodemanager-slave1.out
    slave2: starting nodemanager, logging to /home/ubuntu/hadoop/logs/yarn-ubuntu-nodemanager-slave2.out

2. Testing HDFS by putting files

@on MASTER

#create directories
hadoop fs -mkdir /test_directory

#add a file to the HDFS (random)
hadoop fs -put /etc/hosts /test_directory/

#read files
hadoop fs -cat /test_directory/hosts

3. Running a sample from Hadoop to check MapReduce working

@on MASTER

yarn jar /home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 10 1000000

If you get an output similar to this, your cluster is setup and ready to run codes:

16/06/02 14:36:26 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /networkgeekstuff_output
16/06/02 14:36:32 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/172.31.27.101:8032
16/06/02 14:36:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/06/02 14:36:34 INFO input.FileInputFormat: Total input paths to process : 1
16/06/02 14:36:34 INFO mapreduce.JobSubmitter: number of splits:1
16/06/02 14:36:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464859507107_0002
16/06/02 14:36:34 INFO impl.YarnClientImpl: Submitted application application_1464859507107_0002
16/06/02 14:36:34 INFO mapreduce.Job: The url to track the job: http://hadoopmaster:8088/proxy/application_1464859507107_0002/
16/06/02 14:36:34 INFO mapreduce.Job: Running job: job_1464859507107_0002
16/06/02 14:36:43 INFO mapreduce.Job: Job job_1464859507107_0002 running in uber mode : false
16/06/02 14:36:43 INFO mapreduce.Job:  map 0% reduce 0%
16/06/02 14:36:52 INFO mapreduce.Job:  map 100% reduce 0%
16/06/02 14:37:02 INFO mapreduce.Job:  map 100% reduce 100%
16/06/02 14:37:03 INFO mapreduce.Job: Job job_1464859507107_0002 completed successfully
16/06/02 14:37:03 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=10349090
                FILE: Number of bytes written=20928299
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=5458326
                HDFS: Number of bytes written=717768
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=6794
                Total time spent by all reduces in occupied slots (ms)=6654
                Total time spent by all map tasks (ms)=6794
                Total time spent by all reduce tasks (ms)=6654
                Total vcore-seconds taken by all map tasks=6794
                Total vcore-seconds taken by all reduce tasks=6654
                Total megabyte-seconds taken by all map tasks=6957056
                Total megabyte-seconds taken by all reduce tasks=6813696
        Map-Reduce Framework
                Map input records=124456
                Map output records=901325
                Map output bytes=8546434
                Map output materialized bytes=10349090
                Input split bytes=127
                Combine input records=0
                Combine output records=0
                Reduce input groups=67505
                Reduce shuffle bytes=10349090
                Reduce input records=901325
                Reduce output records=67505
                Spilled Records=1802650
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=220
                CPU time spent (ms)=5600
                Physical memory (bytes) snapshot=327389184
                Virtual memory (bytes) snapshot=1329098752
                Total committed heap usage (bytes)=146931712
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=5458199
        File Output Format Counters 
                Bytes Written=717768