Spark

SparkContext

Src: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SparkContext.html

Reading Files

https://stackoverflow.com/questions/52665353/difference-between-sc-textfile-and-spark-read-text-in-spark

Execution Error

Pyspark exit code 13

Attempting to execute pyspark job in cluster spark-submit --deploy-mode cluster app.py but getting the following error

Application application_1549737803403_0002 failed 2 times due to AM Container for appattempt_1549737803403_0002_000002 exited with exitCode: 13 

Check whether Master mode set to local in code i.e.

sc = SparkContext(master="local", appName="first app")

Remove the master initialization and let --deploy-mode indicate mode i.e.

sc = SparkContext(appName="first app")