Spark Logging - keshavbaweja-git/guides GitHub Wiki

Configure logback as the logging library for Spark

  1. Add required dependencies
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j-api.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>${logback-classic.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>${slf4j-api.version}</version>
            <scope>provided</scope>
        </dependency>
  1. add following libraries under "libs"
  1. Configure driver and executor classpaths
$SPARK_HOME/bin/spark-submit \
--master $SPARK_MASTER  \
--deploy-mode "$7" \
--supervise \
--executor-memory "$8" \
--conf "spark.driver.extraClassPath=$DIR/../libs/*" \
--conf "spark.executor.extraClassPath=$DIR/../libs/*" \
--conf "spark.driver.extraJavaOptions=-Dlogback.configurationFile=$DIR/../config/logback.xml" \
--conf "spark.executor.extraJavaOptions=-Dlogback.configurationFile=$DIR/../config/logback.xml" \
--class com.company.divison.App
⚠️ **GitHub.com Fallback** ⚠️