配置Spark JobHistoryServer - lg1011/SparkLearn GitHub Wiki

1、配置vim spark-defaults.conf
添加如下内容：
spark.master spark://node1:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://ns/history # ns为yarn命名空间

2、配置vim spark-env.sh
添加如下内容：
export SPARK_HISTORY_OPTS=“-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://ns/history”

3、执行spark测试
/opt/spark/bin/spark-submit —class org.apache.spark.examples.SparkPi —master yarn —deploy-mode cluster /opt/spark/examples/jars/spark-examples_2.11-2.2.0.jar 100