Installing Apache Spark on Linux - lmmx/devnotes GitHub Wiki
Spark 1.6.1 (install docs here)
- Recommended to install Scala (the docs themselves don't give much hint)
- Official download and installation docs
- unpack
scala-2.11.8.tgz
to~/opt/scala/
(or/usr/local/share
etc.), then just add that to exports in.bashrc
/~/.bash_profile
- Download Spark, also add to terminal exports, build it
- comes with self-contained Maven installation
- example:
build/mvn --force -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
- I added
--force
to the example, as my Maven is 2.2, requires 3.3+
- I added
- example:
- comes with self-contained Maven installation
Takes about 15 mins to build:
...
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 42.246 s]
[INFO] Spark Project Test Tags ............................ SUCCESS [ 23.970 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 22.225 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 10.819 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.706 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 11.760 s]
[INFO] Spark Project Core ................................. SUCCESS [02:51 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 4.284 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 14.109 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 36.347 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 54.267 s]
[INFO] Spark Project SQL .................................. SUCCESS [01:12 min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:24 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 2.073 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:03 min]
[INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 10.269 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 9.335 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 5.938 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 12.814 s]
[INFO] Spark Project Assembly ............................. SUCCESS [01:46 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 7.596 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 9.086 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 11.412 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 3.722 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 28.518 s]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 8.000 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 7.664 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [ 12.636 s]
[INFO] Spark Project Examples ............................. SUCCESS [02:25 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 6.716 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:47 min
[INFO] Finished at: 2016-06-25T00:50:09+01:00
[INFO] Final Memory: 90M/1373M
[INFO] ------------------------------------------------------------------------