[spark] installation and test on Ubuntu, docker, OS X - dsindex/blog GitHub Wiki

• https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get
• version : java 7

• http://www.scala-lang.org/download/2.10.4.html
• version : scala 2.10.4

• http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
  • single node에서 가상으로 distributed cluster를 구성
• 모든 설정이 끝난 이후, 다시 docker run

docker run -i -t -p 9001:9001 -p 50070:50070 -p 9000:9000 -p 8088:8088 -p 8089:8089 image_name
/etc/init.d/ssh start
/root/hadoop-2.7.2/sbin/start-dfs.sh
/root/hadoop-2.7.2/sbin/start-yarn.sh

• dfs namenode interface : http://192.168.99.100:50070/
• yarn resource manager interface : http://192.168.99.100:8088/
• virtual box에서 port forwarding
  • 이걸 설정해야 OS X host ip로 web interface에 접근이 가능하다.
• 실제 cluster 전체 설치가 필요한 경우 : cluster setup

• http://why-not-learn-something.blogspot.kr/2015/06/spark-installation-pseudo.html
• http://mbonaci.github.io/mbo-spark/
• http://spark.apache.org/docs/latest/spark-standalone.html

/root/spark-1.6.1-bin-without-hadoop/sbin/start-master.sh
/root/spark-1.6.1-bin-without-hadoop/sbin/start-slave.sh spark://localhost:7077

• spark master interface : http://192.168.99.100:8089/
• wordcount example

hadoop fs -put hadoop-2.7.2/etc/hadoop/ ./
hadoop fs -move hadoop input
spark-submit --master spark://localhost:7077 --total-executor-cores 2 --executor-memory 512M  wordcount.py -f input
hadoop fs -ls ./output