Partition - satyamsingh1004/spark GitHub Wiki

Partition Type

  • Input

  • Output

  • Shuffle

image

Partition size

  • Default - 128 MB
  • The number of partitions in Spark executors equals sql.shuffle.partitions if there is at least one wide transformation in the ETL. If only narrow transformations are applied, the number of partitions would match the number created when reading the file.