Install Java

sudo apt-get install default-jdk -y

Download Spark

wget https://ftp.jaist.ac.jp/pub/apache/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz

Extract Spark

tar xvfz spark-3.1.1-bin-hadoop3.2.tgz

Remove the installer file (.tgz)

rm spark-3.1.1-bin-hadoop3.2.tgz

Change the folder permission to 755

chmod -R 755 spark-3.1.1-bin-hadoop3.2/

Add SPARK_HOME

nano ~/.bashrc

At the end of the file, add these lines

SPARK_HOME=/home/arwan_khoiruddin/spark-3.1.1-bin-hadoop3.2/
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Save the file, then restart the bashrc using this command

source ~/.bashrc

Test if spark is correctly installed

spark-shell

It should bring you this output

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/arwan_khoiruddin/spark-3.1.1-bin
-hadoop3.2/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/03/19 10:55:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-ja
va classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://spark.asia-southeast2-a.c.arwproj.internal:4040
Spark context available as 'sc' (master = local[*], app id = local-1616151342942).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.1
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 11.0.10)
Type in expressions to have them evaluated.
Type :help for more information.
scala>