Spark JIRAs - animeshtrivedi/notes GitHub Wiki

Vectorized ORC reader

https://issues.apache.org/jira/browse/SPARK-16060

ORC improvement in Apache Spark 2.3, https://dataworkssummit.com/berlin-2018/session/orc-improvement-in-apache-spark-2-3/

Feature parity for ORC with Parquet

https://issues.apache.org/jira/browse/SPARK-20901

Data source API V2: [SPARK-15689][SPARK-22386]

https://spark.apache.org/releases/spark-release-2-3-0.html

An experimental API for plugging in new data sources in Spark. The new API attempts to address several limitations of the V1 API and aims to facilitate development of high performant, easy-to-maintain, and extensible external data sources. Note that this API is still undergoing active development and breaking changes should be expected.

https://issues.apache.org/jira/browse/SPARK-15689 https://issues.apache.org/jira/browse/SPARK-22386