现记录自己配置过程中遇到的问题，至于java的环境配置在前面已有：ＭＡＣ中ＳＳＨ使用：

http://blog.csdn.net/ljfrocky/article/details/46280455

安装hadoop 2.5.2版本：

Hadoop三种安装模式:讲述这几种模式之间的区别 http://blog.csdn.net/liumm0000/article/details/13408855

当安装伪分布式时配置core-site.xml, hdfs-site.xml后执行 ssh localhost 测试as below:

fuyingdeMacBook-Air:hadoop-2.5.2 fuying$ ssh localhost ssh: connect to host localhost port 22: Connection refused fuyingdeMacBook-Air:hadoop-2.5.2 fuying$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa Generating public/private dsa key pair. Created directory '/Users/fuying/.ssh'. Your identification has been saved in /Users/fuying/.ssh/id_dsa. Your public key has been saved in /Users/fuying/.ssh/id_dsa.pub. The key fingerprint is: e2:d5:c6:d9:e3:07:e8:09:51:cd:52:21:12:b8:ee:4e [email protected] The key's randomart image is: +--[ DSA 1024]----+ | .o.o+o. | | . o..o | | .. . | | . + + | | .. S * + | | ..o + o o | | .E o . . | | .. . | | .. | +-----------------+ fuyingdeMacBook-Air:hadoop-2.5.2 fuying$

在官网上讲述了三种安装模式：单机模式，伪分布模式，第一种最简单， step1: 只需要将安装文件解压放置在/Applications/Utilities/Develop/hadoop-2.5.2即可，再去修改hadoop-env.sh中的JAVA_HOME。 echo ${JAVA_HOME}

step2: 然后执行以下2个实例去验证：

实例1 ： $ mkdir input

$ cp etc/hadoop/*.xml input

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar grep input output 'dfs[a-z.]+'

$ cat output/*

实例2 ： $ mkdir wcinput $ cd wcinput/ $ touch wc.input $ vi wc.input

hadoop yarn

hadoop mapreduce

hadoop hdfs

yarn nodemanager

hadoop recoucemanager

$ cat wc.input

$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount wcinput wcoutput

16/09/27 10:25:26 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 16/09/27 10:25:26 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= java.net.ConnectException: Call From fuyingdeMacBook-Air.local/192.168.2.104 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 40 more

以上错误是由于在core-site.xml中配置了hdfs文件系统，将其先删除； 16/09/27 10:27:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/27 10:27:11 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 16/09/27 10:27:11 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 16/09/27 10:27:11 INFO input.FileInputFormat: Total input paths to process : 1 16/09/27 10:27:11 INFO mapreduce.JobSubmitter: number of splits:1 16/09/27 10:27:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local531909640_0001 16/09/27 10:27:11 WARN conf.Configuration: file:/tmp/hadoop-fuying/mapred/staging/fuying531909640/.staging/job_local531909640_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 16/09/27 10:27:11 WARN conf.Configuration: file:/tmp/hadoop-fuying/mapred/staging/fuying531909640/.staging/job_local531909640_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 16/09/27 10:27:12 WARN conf.Configuration: file:/tmp/hadoop-fuying/mapred/local/localRunner/fuying/job_local531909640_0001/job_local531909640_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 16/09/27 10:27:12 WARN conf.Configuration: file:/tmp/hadoop-fuying/mapred/local/localRunner/fuying/job_local531909640_0001/job_local531909640_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 16/09/27 10:27:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 16/09/27 10:27:12 INFO mapreduce.Job: Running job: job_local531909640_0001 16/09/27 10:27:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null 16/09/27 10:27:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Waiting for map tasks 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Starting task: attempt_local531909640_0001_m_000000_0 16/09/27 10:27:12 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux. 16/09/27 10:27:12 INFO mapred.Task: Using ResourceCalculatorProcessTree : null 16/09/27 10:27:12 INFO mapred.MapTask: Processing split: file:/Applications/Utilities/Develop/hadoop-2.5.2/wcinput/wc.input:0+80 16/09/27 10:27:12 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 16/09/27 10:27:12 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 16/09/27 10:27:12 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 16/09/27 10:27:12 INFO mapred.MapTask: soft limit at 83886080 16/09/27 10:27:12 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 16/09/27 10:27:12 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 16/09/27 10:27:12 INFO mapred.LocalJobRunner: 16/09/27 10:27:12 INFO mapred.MapTask: Starting flush of map output 16/09/27 10:27:12 INFO mapred.MapTask: Spilling map output 16/09/27 10:27:12 INFO mapred.MapTask: bufstart = 0; bufend = 120; bufvoid = 104857600 16/09/27 10:27:12 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214360(104857440); length = 37/6553600 16/09/27 10:27:12 INFO mapred.MapTask: Finished spill 0 16/09/27 10:27:12 INFO mapred.Task: Task:attempt_local531909640_0001_m_000000_0 is done. And is in the process of committing 16/09/27 10:27:12 INFO mapred.LocalJobRunner: map 16/09/27 10:27:12 INFO mapred.Task: Task 'attempt_local531909640_0001_m_000000_0' done. 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local531909640_0001_m_000000_0 16/09/27 10:27:12 INFO mapred.LocalJobRunner: map task executor complete. 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Waiting for reduce tasks 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Starting task: attempt_local531909640_0001_r_000000_0 16/09/27 10:27:12 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux. 16/09/27 10:27:12 INFO mapred.Task: Using ResourceCalculatorProcessTree : null 16/09/27 10:27:12 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4127f9f0 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10 16/09/27 10:27:12 INFO reduce.EventFetcher: attempt_local531909640_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 16/09/27 10:27:12 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local531909640_0001_m_000000_0 decomp: 92 len: 96 to MEMORY 16/09/27 10:27:12 INFO reduce.InMemoryMapOutput: Read 92 bytes from map-output for attempt_local531909640_0001_m_000000_0 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 92, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->92 16/09/27 10:27:12 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning 16/09/27 10:27:12 INFO mapred.LocalJobRunner: 1 / 1 copied. 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 16/09/27 10:27:12 INFO mapred.Merger: Merging 1 sorted segments 16/09/27 10:27:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 83 bytes 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: Merged 1 segments, 92 bytes to disk to satisfy reduce memory limit 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: Merging 1 files, 96 bytes from disk 16/09/27 10:27:12 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 16/09/27 10:27:12 INFO mapred.Merger: Merging 1 sorted segments 16/09/27 10:27:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 83 bytes 16/09/27 10:27:12 INFO mapred.LocalJobRunner: 1 / 1 copied. 16/09/27 10:27:12 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 16/09/27 10:27:12 INFO mapred.Task: Task:attempt_local531909640_0001_r_000000_0 is done. And is in the process of committing 16/09/27 10:27:12 INFO mapred.LocalJobRunner: 1 / 1 copied. 16/09/27 10:27:12 INFO mapred.Task: Task attempt_local531909640_0001_r_000000_0 is allowed to commit now 16/09/27 10:27:12 INFO output.FileOutputCommitter: Saved output of task 'attempt_local531909640_0001_r_000000_0' to file:/Applications/Utilities/Develop/hadoop-2.5.2/wcoutput/_temporary/0/task_local531909640_0001_r_000000 16/09/27 10:27:12 INFO mapred.LocalJobRunner: reduce > reduce 16/09/27 10:27:12 INFO mapred.Task: Task 'attempt_local531909640_0001_r_000000_0' done. 16/09/27 10:27:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local531909640_0001_r_000000_0 16/09/27 10:27:12 INFO mapred.LocalJobRunner: reduce task executor complete. 16/09/27 10:27:13 INFO mapreduce.Job: Job job_local531909640_0001 running in uber mode : false 16/09/27 10:27:13 INFO mapreduce.Job: map 100% reduce 100% 16/09/27 10:27:13 INFO mapreduce.Job: Job job_local531909640_0001 completed successfully 16/09/27 10:27:13 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=541400 FILE: Number of bytes written=1007506 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=5 Map output records=10 Map output bytes=120 Map output materialized bytes=96 Input split bytes=131 Combine input records=10 Combine output records=6 Reduce input groups=6 Reduce shuffle bytes=96 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 Total committed heap usage (bytes)=443547648 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=80 File Output Format Counters Bytes Written=78

cat wcoutput/part-r-00000 hadoop 4 hdfs 1 mapreduce 1 nodemanager 1 recoucemanager 1 yarn 2

第二种方式：按照官网上，

step1:

应在core-site.xml,hdfs-site.xml加上配置属性，不使用其默认的文件系统，而是在指定的配置位置格式化一个HDFS文件系统；其实除了官网上所加的配置，还应修改一些core-default.xml的配置属性，如下： $ mkdir data $ cd data/ $ mkdir tmp $ cd tmp/ $ pwd ［/Applications/Utilities/Develop/hadoop-2.5.2/data/tmp］ fs.defaultFS hdfs://localhost:9000

<property>
    <name>hadoop.tmp.dir</name>
    <value>/Applications/Utilities/Develop/hadoop-2.5.2/data/tmp</value>
</property>

dfs.replication 1

step2:

$ hdfs namenode -format fuyingdeMacBook-Air:hadoop-2.5.2 fuying$ hdfs namenode -format

16/09/27 10:39:17 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = fuyingdeMacBook-Air.local/192.168.2.104 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.5.2 STARTUP_MSG: classpath = /Applications/Utilities/Develop/hadoop-2.5.2/etc/hadoop:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/activation-1.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/asm-3.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/avro-1.7.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-cli-1.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-codec-1.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-digester-1.8.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-el-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-io-2.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-lang-2.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/commons-net-3.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/guava-11.0.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/hadoop-annotations-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/hadoop-auth-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jersey-core-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jersey-json-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jersey-server-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jettison-1.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jetty-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jsch-0.1.42.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jsp-api-2.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/jsr305-1.3.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/junit-4.11.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/log4j-1.2.17.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/paranamer-2.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/servlet-api-2.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/xmlenc-0.52.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/xz-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2-tests.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/common/hadoop-nfs-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/asm-3.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2-tests.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/activation-1.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/asm-3.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/guice-3.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/javax.inject-1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jettison-1.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jline-0.9.94.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/xz-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-api-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-client-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-common-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-common-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2-tests.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar:/Applications/Utilities/Develop/hadoop-2.5.2/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on 2014-11-14T23:45Z STARTUP_MSG: java = 1.7.0_80 / 16/09/27 10:39:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 16/09/27 10:39:17 INFO namenode.NameNode: createNameNode [-format] 16/09/27 10:39:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Formatting using clusterid: CID-4d1d3351-781f-4c21-a5a0-3f56f51d454a 16/09/27 10:39:18 INFO namenode.FSNamesystem: fsLock is fair:true 16/09/27 10:39:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 16/09/27 10:39:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 16/09/27 10:39:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 16/09/27 10:39:18 INFO blockmanagement.BlockManager: The block deletion will start around 2016 九月 27 10:39:18 16/09/27 10:39:18 INFO util.GSet: Computing capacity for map BlocksMap 16/09/27 10:39:18 INFO util.GSet: VM type = 64-bit 16/09/27 10:39:18 INFO util.GSet: 2.0% max memory 1.7 GB = 35.6 MB 16/09/27 10:39:18 INFO util.GSet: capacity = 2^22 = 4194304 entries 16/09/27 10:39:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 16/09/27 10:39:18 INFO blockmanagement.BlockManager: defaultReplication = 1 16/09/27 10:39:18 INFO blockmanagement.BlockManager: maxReplication = 512 16/09/27 10:39:18 INFO blockmanagement.BlockManager: minReplication = 1 16/09/27 10:39:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 16/09/27 10:39:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 16/09/27 10:39:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 16/09/27 10:39:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false 16/09/27 10:39:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 16/09/27 10:39:18 INFO namenode.FSNamesystem: fsOwner = fuying (auth:SIMPLE) 16/09/27 10:39:18 INFO namenode.FSNamesystem: supergroup = supergroup 16/09/27 10:39:18 INFO namenode.FSNamesystem: isPermissionEnabled = true 16/09/27 10:39:18 INFO namenode.FSNamesystem: HA Enabled: false 16/09/27 10:39:18 INFO namenode.FSNamesystem: Append Enabled: true 16/09/27 10:39:18 INFO util.GSet: Computing capacity for map INodeMap 16/09/27 10:39:18 INFO util.GSet: VM type = 64-bit 16/09/27 10:39:18 INFO util.GSet: 1.0% max memory 1.7 GB = 17.8 MB 16/09/27 10:39:18 INFO util.GSet: capacity = 2^21 = 2097152 entries 16/09/27 10:39:18 INFO namenode.NameNode: Caching file names occuring more than 10 times 16/09/27 10:39:18 INFO util.GSet: Computing capacity for map cachedBlocks 16/09/27 10:39:18 INFO util.GSet: VM type = 64-bit 16/09/27 10:39:18 INFO util.GSet: 0.25% max memory 1.7 GB = 4.4 MB 16/09/27 10:39:18 INFO util.GSet: capacity = 2^19 = 524288 entries 16/09/27 10:39:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 16/09/27 10:39:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 16/09/27 10:39:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 16/09/27 10:39:18 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 16/09/27 10:39:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 16/09/27 10:39:18 INFO util.GSet: Computing capacity for map NameNodeRetryCache 16/09/27 10:39:18 INFO util.GSet: VM type = 64-bit 16/09/27 10:39:18 INFO util.GSet: 0.029999999329447746% max memory 1.7 GB = 546.2 KB 16/09/27 10:39:18 INFO util.GSet: capacity = 2^16 = 65536 entries 16/09/27 10:39:18 INFO namenode.NNConf: ACLs enabled? false 16/09/27 10:39:18 INFO namenode.NNConf: XAttrs enabled? true 16/09/27 10:39:18 INFO namenode.NNConf: Maximum size of an xattr: 16384 16/09/27 10:39:18 INFO namenode.FSImage: Allocated new BlockPoolId: BP-75396273-192.168.2.104-1474943958409 16/09/27 10:39:18 INFO common.Storage: Storage directory /Applications/Utilities/Develop/hadoop-2.5.2/data/tmp/dfs/name has been successfully formatted. 16/09/27 10:39:18 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 16/09/27 10:39:18 INFO util.ExitUtil: Exiting with status 0 16/09/27 10:39:18 INFO namenode.NameNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down NameNode at fuyingdeMacBook-Air.local/192.168.2.104 ************************************************************/

step3: $ sbin/hadoop-daemon.sh start namenode

step4: $ sbin/hadoop-daemon.sh start datanode

$ jps $ ls -l logs $ hostname fuyingdeMacBook-Air.local

starting namenode, logging to /Applications/Utilities/Develop/hadoop-2.5.2/logs/hadoop-fuying-namenode-fuyingdeMacBook-Air.local.out fuyingdeMacBook-Air:hadoop-2.5.2 fuying$ sbin/hadoop-daemon.sh start datanode starting datanode, logging to /Applications/Utilities/Develop/hadoop-2.5.2/logs/hadoop-fuying-datanode-fuyingdeMacBook-Air.local.out] 这时可以找开 http://fuyingdemacbook-air.local:50070/dfshealth.html#tab-overview 或 http://localhost:50070/dfshealth.html#tab-overview Utilities-> browse the file system

step5: 可以创建文件去看web UI上的结果 $ bin/hdfs dfs -mkdir -p /user/fuyingTest1 $ bin/hdfs dfs -mkdir -p /user/fupingTest1/mapreduce/wordcount/input

step6: 试着上传文件到上面去 $ bin/hdfs dfs -put wcinput/wc.input /user/fupingTest1/mapreduce/wordcount/input/ $ bin/hdfs dfs -ls /user/fupingTest1/mapreduce/wordcount/input/ $ bin/hdfs dfs -cat /user/fupingTest1/mapreduce/wordcount/input/wc.input

step7: 试着执行HDFS上的文件 $ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/fupingTest1/mapreduce/wordcount/input/ /user/fupingTest1/mapreduce/wordcount/output $ bin/hdfs dfs -cat /user/fupingTest1/mapreduce/output/part*

第三种方式: step1: 修改yarn-env.sh, mapred-env.sh中的JAVA_HOME; step2: 修改配置文件yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

step3: 必要时修改slaves文件 step4: 重命名：mapred-site.xml.template 为 mapred-site.xml，并修改其配置文件按官网 mapreduce.framework.name yarn

step5: $ sbin/yarn-daemon.sh start resourcemanager $ sbin/yarn-daemon.sh start nodemanager $ jps 728 DataNode 1310 ResourceManager 681 NameNode 1380 Jps 1351 NodeManager

这个时候可以通过http://localhost:8088/查看监控界面： step6: 运行测试先删除ＨＤＦＳ上的output文件 $ bin/hdfs dfs -rm -R /user/fupingTest1/mapreduce/wordcount/output/ $ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/fupingTest1/mapreduce/wordcount/input/ /user/fupingTest1/mapreduce/wordcount/output 16/09/27 12:04:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/27 12:04:27 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 16/09/27 12:04:28 INFO input.FileInputFormat: Total input paths to process : 1 16/09/27 12:04:28 INFO mapreduce.JobSubmitter: number of splits:1 16/09/27 12:04:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1474948460665_0001 16/09/27 12:04:29 INFO impl.YarnClientImpl: Submitted application application_1474948460665_0001 16/09/27 12:04:29 INFO mapreduce.Job: The url to track the job: http://192.168.2.104:8088/proxy/application_1474948460665_0001/ 16/09/27 12:04:29 INFO mapreduce.Job: Running job: job_1474948460665_0001 16/09/27 12:04:41 INFO mapreduce.Job: Job job_1474948460665_0001 running in uber mode : false 16/09/27 12:04:41 INFO mapreduce.Job: map 0% reduce 0% 16/09/27 12:04:49 INFO mapreduce.Job: map 100% reduce 0% 16/09/27 12:04:57 INFO mapreduce.Job: map 100% reduce 100% 16/09/27 12:04:57 INFO mapreduce.Job: Job job_1474948460665_0001 completed successfully 16/09/27 12:04:57 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=96 FILE: Number of bytes written=194667 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=218 HDFS: Number of bytes written=66 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5868 Total time spent by all reduces in occupied slots (ms)=4361 Total time spent by all map tasks (ms)=5868 Total time spent by all reduce tasks (ms)=4361 Total vcore-seconds taken by all map tasks=5868 Total vcore-seconds taken by all reduce tasks=4361 Total megabyte-seconds taken by all map tasks=6008832 Total megabyte-seconds taken by all reduce tasks=4465664 Map-Reduce Framework Map input records=5 Map output records=10 Map output bytes=120 Map output materialized bytes=96 Input split bytes=138 Combine input records=10 Combine output records=6 Reduce input groups=6 Reduce shuffle bytes=96 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=65 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=286785536 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=80 File Output Format Counters Bytes Written=66

在监控界面上可以看到关于job的信息 application_1474948460665_0001

完全分布式： http://blog.csdn.net/bluecom24/article/details/39593793 ssh username@ip http://blog.csdn.net/jymn_chen/article/details/39931469 http://blog.csdn.net/wk51920/article/details/51686038

在实际的操作过程中会一步步遇到如下问题：第一点需要找到虚拟机：这里选用的三台分布式分别是： mac以及两台安装在ubuntukylin 14.04上的两个ubuntu系统；为方便使用更改ip地址及主机名： hostname查看主机名 ubuntu上：sudo vi /etc/hostname sudo vi /etc/hosts

127.0.0.1 localhost 10.211.55.3 ubuntu1 192.168.2.105 ** mac 10.211.55.4 ubuntu2

其中常见的vi编辑命令：http://www.cnblogs.com/sophine/archive/2012/04/26/2471188.html 光标移动；

当我们按ESC进入Command模式后，我们可以用下面的一些键位来移动光标；

j 向下移动一行； k 向上移动一行；

h 向左移动一个字符； l 向右移动一个字符；

插入模式（文本的插入）；

i 在光标之前插入； a 在光标之后插入；

I 在光标所在行的行首插入； A 在光标所在行的行末插入；

o 在光标所在的行的上面插入一行； O 在光标所在的行的下面插入一行；

s 删除光标后的一个字符，然后进入插入模式； S 删除光标所在的行，然后进入插入模式；

文本内容的删除操作；

x 一个字符； #x 删除几个字符，#表示数字，比如3x； dw 删除一个单词； #dw 删除几个单词，#用数字表示，比如3dw表示删除三个单词； dd 删除一行； #dd 删除多个行，#代表数字，比如3dd 表示删除光标行及光标的下两行； d$ 删除光标到行尾的内容；

J 清除光标所处的行与上一行之间的空格，把光标行和上一行接在一起；

u 撤消修改或删除操作；

按ESC键返回Command（命令）模式，然后按u键来撤消删除以前的删除或修改；如果您想撤消多个以前的修改或删除操作，请按多按几次u。这和Word的撤消操作没有太大的区别；

step2: 安装ＪＤＫ： http://www.cnblogs.com/savagemorgan/p/3650926.html 编辑 .bashrc 文件。

在终端输入如下命令：

vi ~/.bashrc 在该文件的末尾，加上以上几行代码：

export JAVA_HOME=/opt/Java/jdk/jdk1.7 export CLASSPATH=${JAVA_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH

为了让更改立即生效，请在终端执行如下命令：

source ~/.bashrc

全部选用的JDK1.7 想从一台机拷贝到另一台机：http://www.cnblogs.com/jiangyao/archive/2011/01/26/1945570.html scp file user@ubuntu:path 想用免密码，所以使用ssh mac上：http://blog.csdn.net/jymn_chen/article/details/39931469 但ubuntu上要安装：http://blog.csdn.net/bluecom24/article/details/39593793 sudo apt-get install openssh-client=1:6.6p1-2ubuntu1
sudo apt-get install openssh-server

然后配置免密码登录：http://blog.csdn.net/wk51920/article/details/51686038

但由于一开始设置的三台机的登录名不一样，所以配的时候还遇到了一直要带用户名@主机的情况： http://blog.csdn.net/tragedyxd/article/details/46284949 常用如下几个： rm -d 目录名 #删除一个空目录 rmdir 目录名 #删除一个空目录 rm -r 目录名 #删除一个非空目录 rm 文件名 #删除文件

修改本地登陆用户的 ~/.ssh/config 文件，如果木有的话就自个儿建一个吧，内容如下：

Host theoden user liluo Host fili user liluo Host hostname user name

产生秘钥：

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

此时会在~/.ssh文件夹下出现如下两个文件：id_dsa id_dsa.pub

导入authorized_keys：

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys #保证authorized_keys的chmod为600

测试是否安装成功：

#查看是否有sshd进程 ps -e | grep ssh #尝试登录本地 ssh localhost

远程无密码登陆：

#进入master的.ssh目录 scp authorized_keys wk51920@node1:~/.ssh/authorized_keys_master #wk51920是我目前在node1虚拟机上登录系统的用户名，建议三个系统使用相同的用户名 #进入node1、node2的.ssh目录 cat authorized_keys_master >> authorized_keys

可能遇到的问题： http://blog.csdn.net/zhangt85/article/details/42078347

在MAC上安装hadoop - yingziaiai/SetupEnv GitHub Wiki

step1:

step2:

⚠️ GitHub.com Fallback ⚠️

在MAC上安装hadoop - yingziaiai/SetupEnv GitHub Wiki

step1:

step2:

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️