Installing Apache Spark on Linux - lmmx/devnotes GitHub Wiki

Spark 1.6.1 (install docs here)

  • Recommended to install Scala (the docs themselves don't give much hint)
  • Download Spark, also add to terminal exports, build it
    • comes with self-contained Maven installation
      • example: build/mvn --force -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
        • I added --force to the example, as my Maven is 2.2, requires 3.3+

Takes about 15 mins to build:

...
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [ 42.246 s]
[INFO] Spark Project Test Tags ............................ SUCCESS [ 23.970 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 22.225 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 10.819 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  6.706 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 11.760 s]
[INFO] Spark Project Core ................................. SUCCESS [02:51 min]
[INFO] Spark Project Bagel ................................ SUCCESS [  4.284 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 14.109 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 36.347 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 54.267 s]
[INFO] Spark Project SQL .................................. SUCCESS [01:12 min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:24 min]
[INFO] Spark Project Tools ................................ SUCCESS [  2.073 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:03 min]
[INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 10.269 s]
[INFO] Spark Project REPL ................................. SUCCESS [  9.335 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [  5.938 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 12.814 s]
[INFO] Spark Project Assembly ............................. SUCCESS [01:46 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [  7.596 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [  9.086 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 11.412 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  3.722 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 28.518 s]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [  8.000 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [  7.664 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [ 12.636 s]
[INFO] Spark Project Examples ............................. SUCCESS [02:25 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  6.716 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:47 min
[INFO] Finished at: 2016-06-25T00:50:09+01:00
[INFO] Final Memory: 90M/1373M
[INFO] ------------------------------------------------------------------------