Apache Spark™ Learning Resources - datacouch-io/spark-java GitHub Wiki

Welcome to the Apache Spark™ Learning Resources repository! This project is a collection of materials designed to help you understand and work with Apache Spark™ efficiently. Below is an index of the contents available:

Table of Contents

• Introduction to Spark

  • Get started with Apache Spark™ and understand its key features.

• Resilient Distributed Dataset (RDD)

  • Dive deep into the concept of RDD, a fundamental data structure in Spark.

• Spark Context and Spark Session

  • Understand the Spark Context and Spark Session, crucial components in Spark applications.

• RDD Operations

  • Explore various operations you can perform on RDDs, such as transformations and actions.

• Exploring Data Using RDD Operations

  • Learn the essentials of RDD, including its creation and transformation.

• Pair RDD Operations

  • Delve into Pair RDD operations, which are essential for key-value pair data processing.

• Spark on YARN

• Operations On Multiple RDDs

  • In this lab, we will look at several transformations and examine the optimizations and visualise with DAG.

• Corrupt Data Handling

• Advanced Transformations

• Advanced Actions

• Accumulators

• Broadcast

• Custom Partitioner

• Spark Partitioning & Partition

• Spark SQL

  • Dataframe and Spark SQL
  • Spark SQL Fundamentals
    • Assignment - House Price Problem
  • Basic dataframe with AWS S3
  • DataFrame Transformations
  • Dataset Typed API
  • Splitting Text Data
  • Seeing Catalyst at Work
  • Spark Joins
    • Assignment ‐ Joins and Broadcast
  • Cost-Based Optimization (CBO)

• Advanced DataFrame

  • Hive with Spark Overview
  • Integrating Hive and Spark
  • SQL Tables and Views
  • User-Defined Functions
  • Optimizing and Tuning Spark Applications

• Delta Lake

These resources aim to provide you with a comprehensive understanding of Apache Spark™, from its basics to more advanced topics. Feel free to explore each section as you embark on your journey to master Spark. Happy learning!