Hadoop 2.6 1st run troubleshooting notes - mikec964/chelmbigstock GitHub Wiki
If you're following the steps to install Hadoop 2.6.0 in pseudo-distributed mode, here are some troubleshooting tips for the quick test. The example has two basic steps:
-
Load files
bin/hdfs dfs -put etc/hadoop input -
Run jar
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
In my experience, the example fails because the first step loads files and a directory into the input folder. The directory happens to be named hadoop, but the real problem is that the example jar is built to process files and hangs when it hits the directory.
To fix this, after step 1 and before step 2 do two things:
-
Delete the hadoop directory:
bin/hadoop dfs -rm -r input/hadoop -
Delete the output directory:
bin/hadoop dfs -rm -r output
This should give you a successful run.
If you're still having problems, read through the output. You'll see:
No job jar file set
WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
I don't know about this one.
Native Hadoop library
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
If you're on a Mac, ignore this for now. This doesn't change the operation, though non-native may run slower.
Input paths to process
INFO input.FileInputFormat: Total input paths to process : 31
INFO mapreduce.JobSubmitter: number of splits:31
This should match the number of files you uploaded into the input directory.
URL to track job
INFO mapreduce.Job: The url to track the job: http://localhost:8080/
Per Map
15/04/04 13:59:28 INFO mapred.LocalJobRunner: Starting task: attempt_local1899262519_0001_m_000000_0
15/04/04 13:59:28 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 13:59:28 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 13:59:28 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/log4j.properties:0+11291
15/04/04 13:59:28 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 13:59:28 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 13:59:28 INFO mapred.MapTask: soft limit at 83886080
15/04/04 13:59:28 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 13:59:28 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 13:59:28 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 13:59:28 INFO mapred.LocalJobRunner:
15/04/04 13:59:28 INFO mapred.MapTask: Starting flush of map output
15/04/04 13:59:28 INFO mapred.MapTask: Spilling map output
15/04/04 13:59:28 INFO mapred.MapTask: bufstart = 0; bufend = 279; bufvoid = 104857600
15/04/04 13:59:28 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
15/04/04 13:59:28 INFO mapred.MapTask: Finished spill 0
15/04/04 13:59:28 INFO mapred.Task: Task:attempt_local1899262519_0001_m_000000_0 is done. And is in the process of committing
15/04/04 13:59:28 INFO mapred.LocalJobRunner: map
15/04/04 13:59:28 INFO mapred.Task: Task 'attempt_local1899262519_0001_m_000000_0' done.
15/04/04 13:59:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local1899262519_0001_m_000000_0
Path is not a file
WARN mapred.LocalJobRunner: job_local1899262519_0001
java.lang.Exception: java.io.FileNotFoundException: Path is not a file: /user/hadoop/input/hadoop
I think this happens when you have a folder inside the input directory.
FileAlreadyExistsException: Output
INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/hadoop/output already exists
You can fix this with bin/hadooop dfs -rm -r hadoop/output. Do this before each run, or direct your output to a different directory (like output2).
Sample of output file
To access:
bin/hadoop dfs -get output output
cat output/*
6 dfs.audit.logger
4 dfs.class
3 dfs.server.namenode.
2 dfs.audit.log.maxbackupindex
2 dfs.datanode.data.dir
2 dfs.period
2 dfs.blocksize
2 dfs.namenode.name.dir
2 dfs.audit.log.maxfilesize
1 dfsmetrics.log
1 dfsadmin
1 dfs.sh
1 dfs.servers
1 dfs.replication
1 dfs.name.dir
1 dfs.data.dir
1 dfs.block.size
1 dfs.file
Sample of working output
Robin:hadoop-2.6.0 hadoop$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
15/04/04 15:18:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/04 15:18:13 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/04/04 15:18:13 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/04/04 15:18:14 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
15/04/04 15:18:14 INFO input.FileInputFormat: Total input paths to process : 30
15/04/04 15:18:14 INFO mapreduce.JobSubmitter: number of splits:30
15/04/04 15:18:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local524957757_0001
15/04/04 15:18:14 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/04/04 15:18:14 INFO mapreduce.Job: Running job: job_local524957757_0001
15/04/04 15:18:14 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/04/04 15:18:14 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000000_0
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Waiting for map tasks
15/04/04 15:18:15 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:15 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:15 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/log4j.properties:0+11291
15/04/04 15:18:15 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:15 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:15 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:15 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:15 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:15 INFO mapred.LocalJobRunner:
15/04/04 15:18:15 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:15 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:15 INFO mapred.MapTask: bufstart = 0; bufend = 279; bufvoid = 104857600
15/04/04 15:18:15 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
15/04/04 15:18:15 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:15 INFO mapred.Task: Task:attempt_local524957757_0001_m_000000_0 is done. And is in the process of committing
15/04/04 15:18:15 INFO mapred.LocalJobRunner: map
15/04/04 15:18:15 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000000_0' done.
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000000_0
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000001_0
15/04/04 15:18:15 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:15 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:15 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-policy.xml:0+9683
15/04/04 15:18:15 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:15 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:15 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:15 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:15 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:15 INFO mapred.LocalJobRunner:
15/04/04 15:18:15 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:15 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:15 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
15/04/04 15:18:15 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
15/04/04 15:18:15 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:15 INFO mapred.Task: Task:attempt_local524957757_0001_m_000001_0 is done. And is in the process of committing
15/04/04 15:18:15 INFO mapred.LocalJobRunner: map
15/04/04 15:18:15 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000001_0' done.
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000001_0
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000002_0
15/04/04 15:18:15 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:15 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:15 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-site.xml:0+5511
15/04/04 15:18:15 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:15 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:15 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:15 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:15 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:15 INFO mapred.LocalJobRunner:
15/04/04 15:18:15 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:15 INFO mapred.Task: Task:attempt_local524957757_0001_m_000002_0 is done. And is in the process of committing
15/04/04 15:18:15 INFO mapred.LocalJobRunner: map
15/04/04 15:18:15 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000002_0' done.
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000002_0
15/04/04 15:18:15 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000003_0
15/04/04 15:18:15 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:15 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:15 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-env.sh:0+4567
15/04/04 15:18:16 INFO mapreduce.Job: Job job_local524957757_0001 running in uber mode : false
15/04/04 15:18:16 INFO mapreduce.Job: map 100% reduce 0%
15/04/04 15:18:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:16 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:16 INFO mapred.LocalJobRunner:
15/04/04 15:18:16 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:16 INFO mapred.Task: Task:attempt_local524957757_0001_m_000003_0 is done. And is in the process of committing
15/04/04 15:18:16 INFO mapred.LocalJobRunner: map
15/04/04 15:18:16 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000003_0' done.
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000003_0
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000004_0
15/04/04 15:18:16 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:16 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-env.sh:0+4480
15/04/04 15:18:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:16 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:16 INFO mapred.LocalJobRunner:
15/04/04 15:18:16 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:16 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufend = 50; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
15/04/04 15:18:16 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:16 INFO mapred.Task: Task:attempt_local524957757_0001_m_000004_0 is done. And is in the process of committing
15/04/04 15:18:16 INFO mapred.LocalJobRunner: map
15/04/04 15:18:16 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000004_0' done.
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000004_0
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000005_0
15/04/04 15:18:16 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:16 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/capacity-scheduler.xml:0+4436
15/04/04 15:18:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:16 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:16 INFO mapred.LocalJobRunner:
15/04/04 15:18:16 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:16 INFO mapred.Task: Task:attempt_local524957757_0001_m_000005_0 is done. And is in the process of committing
15/04/04 15:18:16 INFO mapred.LocalJobRunner: map
15/04/04 15:18:16 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000005_0' done.
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000005_0
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000006_0
15/04/04 15:18:16 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:16 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-queues.xml.template:0+4113
15/04/04 15:18:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:16 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:16 INFO mapred.LocalJobRunner:
15/04/04 15:18:16 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:16 INFO mapred.Task: Task:attempt_local524957757_0001_m_000006_0 is done. And is in the process of committing
15/04/04 15:18:16 INFO mapred.LocalJobRunner: map
15/04/04 15:18:16 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000006_0' done.
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000006_0
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000007_0
15/04/04 15:18:16 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:16 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-env.cmd:0+3670
15/04/04 15:18:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:16 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:16 INFO mapred.LocalJobRunner:
15/04/04 15:18:16 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:16 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:16 INFO mapred.MapTask: bufstart = 0; bufend = 50; bufvoid = 104857600
15/04/04 15:18:16 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
15/04/04 15:18:16 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:16 INFO mapred.Task: Task:attempt_local524957757_0001_m_000007_0 is done. And is in the process of committing
15/04/04 15:18:16 INFO mapred.LocalJobRunner: map
15/04/04 15:18:16 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000007_0' done.
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000007_0
15/04/04 15:18:16 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000008_0
15/04/04 15:18:16 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:16 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:16 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-acls.xml:0+3523
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000008_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000008_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000008_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000009_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-metrics2.properties:0+2598
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000009_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000009_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000009_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000010_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-metrics.properties:0+2490
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufend = 170; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
15/04/04 15:18:17 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000010_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000010_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000010_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000011_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/ssl-client.xml.example:0+2316
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000011_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000011_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000011_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000012_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/ssl-server.xml.example:0+2268
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000012_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000012_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000012_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000013_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-env.cmd:0+2237
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:17 INFO mapred.LocalJobRunner:
15/04/04 15:18:17 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:17 INFO mapred.Task: Task:attempt_local524957757_0001_m_000013_0 is done. And is in the process of committing
15/04/04 15:18:17 INFO mapred.LocalJobRunner: map
15/04/04 15:18:17 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000013_0' done.
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000013_0
15/04/04 15:18:17 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000014_0
15/04/04 15:18:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hdfs-site.xml:0+1971
15/04/04 15:18:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:17 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufend = 268; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
15/04/04 15:18:18 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000014_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000014_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000014_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000015_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-log4j.properties:0+1657
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000015_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000015_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000015_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000016_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-log4j.properties:0+1631
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000016_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000016_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000016_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000017_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-site.xml:0+1608
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000017_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000017_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000017_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000018_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-env.sh:0+1449
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000018_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000018_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000018_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000019_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-env.sh:0+1383
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000019_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000019_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000019_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000020_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/configuration.xsl:0+1335
15/04/04 15:18:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:18 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:18 INFO mapred.LocalJobRunner:
15/04/04 15:18:18 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:18 INFO mapred.Task: Task:attempt_local524957757_0001_m_000020_0 is done. And is in the process of committing
15/04/04 15:18:18 INFO mapred.LocalJobRunner: map
15/04/04 15:18:18 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000020_0' done.
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000020_0
15/04/04 15:18:18 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000021_0
15/04/04 15:18:18 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:18 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:18 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-env.sh:0+1325
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000021_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000021_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000021_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000022_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/core-site.xml:0+1143
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000022_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapreduce.Job: map 73% reduce 0%
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000022_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000022_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000023_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-env.cmd:0+938
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000023_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000023_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000023_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000024_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-site.xml.template:0+758
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000024_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000024_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000024_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000025_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-site.xml:0+690
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000025_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000025_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000025_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000026_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-site.xml:0+620
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000026_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000026_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000026_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000027_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/container-executor.cfg:0+318
15/04/04 15:18:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:19 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:19 INFO mapred.LocalJobRunner:
15/04/04 15:18:19 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:19 INFO mapred.Task: Task:attempt_local524957757_0001_m_000027_0 is done. And is in the process of committing
15/04/04 15:18:19 INFO mapred.LocalJobRunner: map
15/04/04 15:18:19 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000027_0' done.
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000027_0
15/04/04 15:18:19 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000028_0
15/04/04 15:18:19 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:19 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:19 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-signature.secret:0+21
15/04/04 15:18:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:20 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:20 INFO mapred.LocalJobRunner:
15/04/04 15:18:20 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:20 INFO mapred.Task: Task:attempt_local524957757_0001_m_000028_0 is done. And is in the process of committing
15/04/04 15:18:20 INFO mapred.LocalJobRunner: map
15/04/04 15:18:20 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000028_0' done.
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000028_0
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_m_000029_0
15/04/04 15:18:20 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:20 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:20 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/slaves:0+10
15/04/04 15:18:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:20 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:20 INFO mapreduce.Job: map 100% reduce 0%
15/04/04 15:18:20 INFO mapred.LocalJobRunner:
15/04/04 15:18:20 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:20 INFO mapred.Task: Task:attempt_local524957757_0001_m_000029_0 is done. And is in the process of committing
15/04/04 15:18:20 INFO mapred.LocalJobRunner: map
15/04/04 15:18:20 INFO mapred.Task: Task 'attempt_local524957757_0001_m_000029_0' done.
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_m_000029_0
15/04/04 15:18:20 INFO mapred.LocalJobRunner: map task executor complete.
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Waiting for reduce tasks
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Starting task: attempt_local524957757_0001_r_000000_0
15/04/04 15:18:20 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:20 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:20 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@36d0925d
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10
15/04/04 15:18:20 INFO reduce.EventFetcher: attempt_local524957757_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000014_0 decomp: 204 len: 208 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 204 bytes from map-output for attempt_local524957757_0001_m_000014_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 204, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->204
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000002_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 204, usedMemory ->206
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000026_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000026_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 206, usedMemory ->208
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000025_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000025_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 208, usedMemory ->210
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000001_0 decomp: 21 len: 25 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local524957757_0001_m_000001_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 5, commitMemory -> 210, usedMemory ->231
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000015_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000015_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 231, usedMemory ->233
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000012_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000012_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 233, usedMemory ->235
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000000_0 decomp: 135 len: 139 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 135 bytes from map-output for attempt_local524957757_0001_m_000000_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 135, inMemoryMapOutputs.size() -> 8, commitMemory -> 235, usedMemory ->370
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000013_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000013_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 370, usedMemory ->372
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000027_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000027_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 372, usedMemory ->374
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000017_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000017_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 374, usedMemory ->376
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000005_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 376, usedMemory ->378
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000029_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000029_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 378, usedMemory ->380
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000028_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000028_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 380, usedMemory ->382
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000004_0 decomp: 29 len: 33 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 29 bytes from map-output for attempt_local524957757_0001_m_000004_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 29, inMemoryMapOutputs.size() -> 15, commitMemory -> 382, usedMemory ->411
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000018_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000018_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 411, usedMemory ->413
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000016_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000016_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 17, commitMemory -> 413, usedMemory ->415
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000003_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 18, commitMemory -> 415, usedMemory ->417
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000008_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 19, commitMemory -> 417, usedMemory ->419
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000020_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000020_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 20, commitMemory -> 419, usedMemory ->421
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000021_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000021_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 21, commitMemory -> 421, usedMemory ->423
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000007_0 decomp: 29 len: 33 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 29 bytes from map-output for attempt_local524957757_0001_m_000007_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 29, inMemoryMapOutputs.size() -> 22, commitMemory -> 423, usedMemory ->452
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000019_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000019_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 23, commitMemory -> 452, usedMemory ->454
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000006_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 24, commitMemory -> 454, usedMemory ->456
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000011_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000011_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 25, commitMemory -> 456, usedMemory ->458
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000024_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000024_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 26, commitMemory -> 458, usedMemory ->460
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000022_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000022_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 27, commitMemory -> 460, usedMemory ->462
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000010_0 decomp: 109 len: 113 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 109 bytes from map-output for attempt_local524957757_0001_m_000010_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 109, inMemoryMapOutputs.size() -> 28, commitMemory -> 462, usedMemory ->571
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000023_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000023_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 29, commitMemory -> 571, usedMemory ->573
15/04/04 15:18:20 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local524957757_0001_m_000009_0 decomp: 2 len: 6 to MEMORY
15/04/04 15:18:20 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local524957757_0001_m_000009_0
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 30, commitMemory -> 573, usedMemory ->575
15/04/04 15:18:20 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/04/04 15:18:20 INFO mapred.LocalJobRunner: 30 / 30 copied.
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: finalMerge called with 30 in-memory map-outputs and 0 on-disk map-outputs
15/04/04 15:18:20 INFO mapred.Merger: Merging 30 sorted segments
15/04/04 15:18:20 INFO mapred.Merger: Down to the last merge-pass, with 6 segments left of total size: 418 bytes
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: Merged 30 segments, 575 bytes to disk to satisfy reduce memory limit
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: Merging 1 files, 521 bytes from disk
15/04/04 15:18:20 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
15/04/04 15:18:20 INFO mapred.Merger: Merging 1 sorted segments
15/04/04 15:18:20 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 486 bytes
15/04/04 15:18:20 INFO mapred.LocalJobRunner: 30 / 30 copied.
15/04/04 15:18:20 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
15/04/04 15:18:20 INFO mapred.Task: Task:attempt_local524957757_0001_r_000000_0 is done. And is in the process of committing
15/04/04 15:18:20 INFO mapred.LocalJobRunner: 30 / 30 copied.
15/04/04 15:18:20 INFO mapred.Task: Task attempt_local524957757_0001_r_000000_0 is allowed to commit now
15/04/04 15:18:20 INFO output.FileOutputCommitter: Saved output of task 'attempt_local524957757_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-792679216/_temporary/0/task_local524957757_0001_r_000000
15/04/04 15:18:20 INFO mapred.LocalJobRunner: reduce > reduce
15/04/04 15:18:20 INFO mapred.Task: Task 'attempt_local524957757_0001_r_000000_0' done.
15/04/04 15:18:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local524957757_0001_r_000000_0
15/04/04 15:18:20 INFO mapred.LocalJobRunner: reduce task executor complete.
15/04/04 15:18:21 INFO mapreduce.Job: map 100% reduce 100%
15/04/04 15:18:21 INFO mapreduce.Job: Job job_local524957757_0001 completed successfully
15/04/04 15:18:21 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=1367087
FILE: Number of bytes written=7973809
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1877616
HDFS: Number of bytes written=655
HDFS: Number of read operations=1117
HDFS: Number of large read operations=0
HDFS: Number of write operations=33
Map-Reduce Framework
Map input records=2158
Map output records=34
Map output bytes=834
Map output materialized bytes=695
Input split bytes=3654
Combine input records=34
Combine output records=20
Reduce input groups=18
Reduce shuffle bytes=695
Reduce input records=20
Reduce output records=18
Spilled Records=40
Shuffled Maps =30
Failed Shuffles=0
Merged Map outputs=30
GC time elapsed (ms)=926
Total committed heap usage (bytes)=11960057856
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=80040
File Output Format Counters
Bytes Written=655
15/04/04 15:18:21 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
15/04/04 15:18:21 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
15/04/04 15:18:21 INFO input.FileInputFormat: Total input paths to process : 1
15/04/04 15:18:21 INFO mapreduce.JobSubmitter: number of splits:1
15/04/04 15:18:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1106183829_0002
15/04/04 15:18:21 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/04/04 15:18:21 INFO mapreduce.Job: Running job: job_local1106183829_0002
15/04/04 15:18:21 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/04/04 15:18:21 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
15/04/04 15:18:21 INFO mapred.LocalJobRunner: Waiting for map tasks
15/04/04 15:18:21 INFO mapred.LocalJobRunner: Starting task: attempt_local1106183829_0002_m_000000_0
15/04/04 15:18:21 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:21 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:21 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-792679216/part-r-00000:0+655
15/04/04 15:18:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/04/04 15:18:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/04/04 15:18:22 INFO mapred.MapTask: soft limit at 83886080
15/04/04 15:18:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/04/04 15:18:22 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/04/04 15:18:22 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/04/04 15:18:22 INFO mapred.LocalJobRunner:
15/04/04 15:18:22 INFO mapred.MapTask: Starting flush of map output
15/04/04 15:18:22 INFO mapred.MapTask: Spilling map output
15/04/04 15:18:22 INFO mapred.MapTask: bufstart = 0; bufend = 425; bufvoid = 104857600
15/04/04 15:18:22 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214328(104857312); length = 69/6553600
15/04/04 15:18:22 INFO mapred.MapTask: Finished spill 0
15/04/04 15:18:22 INFO mapred.Task: Task:attempt_local1106183829_0002_m_000000_0 is done. And is in the process of committing
15/04/04 15:18:22 INFO mapred.LocalJobRunner: map
15/04/04 15:18:22 INFO mapred.Task: Task 'attempt_local1106183829_0002_m_000000_0' done.
15/04/04 15:18:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local1106183829_0002_m_000000_0
15/04/04 15:18:22 INFO mapred.LocalJobRunner: map task executor complete.
15/04/04 15:18:22 INFO mapred.LocalJobRunner: Waiting for reduce tasks
15/04/04 15:18:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1106183829_0002_r_000000_0
15/04/04 15:18:22 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/04/04 15:18:22 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/04/04 15:18:22 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@33b1594e
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10
15/04/04 15:18:22 INFO reduce.EventFetcher: attempt_local1106183829_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
15/04/04 15:18:22 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1106183829_0002_m_000000_0 decomp: 463 len: 467 to MEMORY
15/04/04 15:18:22 INFO reduce.InMemoryMapOutput: Read 463 bytes from map-output for attempt_local1106183829_0002_m_000000_0
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 463, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->463
15/04/04 15:18:22 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/04/04 15:18:22 INFO mapred.LocalJobRunner: 1 / 1 copied.
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
15/04/04 15:18:22 INFO mapred.Merger: Merging 1 sorted segments
15/04/04 15:18:22 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 453 bytes
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: Merged 1 segments, 463 bytes to disk to satisfy reduce memory limit
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: Merging 1 files, 467 bytes from disk
15/04/04 15:18:22 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
15/04/04 15:18:22 INFO mapred.Merger: Merging 1 sorted segments
15/04/04 15:18:22 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 453 bytes
15/04/04 15:18:22 INFO mapred.LocalJobRunner: 1 / 1 copied.
15/04/04 15:18:22 INFO mapred.Task: Task:attempt_local1106183829_0002_r_000000_0 is done. And is in the process of committing
15/04/04 15:18:22 INFO mapred.LocalJobRunner: 1 / 1 copied.
15/04/04 15:18:22 INFO mapred.Task: Task attempt_local1106183829_0002_r_000000_0 is allowed to commit now
15/04/04 15:18:22 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1106183829_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output/_temporary/0/task_local1106183829_0002_r_000000
15/04/04 15:18:22 INFO mapred.LocalJobRunner: reduce > reduce
15/04/04 15:18:22 INFO mapred.Task: Task 'attempt_local1106183829_0002_r_000000_0' done.
15/04/04 15:18:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local1106183829_0002_r_000000_0
15/04/04 15:18:22 INFO mapred.LocalJobRunner: reduce task executor complete.
15/04/04 15:18:22 INFO mapreduce.Job: Job job_local1106183829_0002 running in uber mode : false
15/04/04 15:18:22 INFO mapreduce.Job: map 100% reduce 100%
15/04/04 15:18:22 INFO mapreduce.Job: Job job_local1106183829_0002 completed successfully
15/04/04 15:18:22 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=137664
FILE: Number of bytes written=1021869
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=161390
HDFS: Number of bytes written=1627
HDFS: Number of read operations=161
HDFS: Number of large read operations=0
HDFS: Number of write operations=16
Map-Reduce Framework
Map input records=18
Map output records=18
Map output bytes=425
Map output materialized bytes=467
Input split bytes=131
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=467
Reduce input records=18
Reduce output records=18
Spilled Records=36
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=4
Total committed heap usage (bytes)=784334848
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=655
File Output Format Counters
Bytes Written=317