-
- Downloads
[SPARK-11955][SQL] Mark optional fields in merging schema for safely pushdowning filters in Parquet
JIRA: https://issues.apache.org/jira/browse/SPARK-11955 Currently we simply skip pushdowning filters in parquet if we enable schema merging. However, we can actually mark particular fields in merging schema for safely pushdowning filters in parquet. Author: Liang-Chi Hsieh <viirya@appier.com> Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #9940 from viirya/safe-pushdown-parquet-filters.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala 5 additions, 0 deletions.../src/main/scala/org/apache/spark/sql/types/Metadata.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala 28 additions, 6 deletions...rc/main/scala/org/apache/spark/sql/types/StructType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 10 additions, 4 deletions...test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala 26 additions, 11 deletions...rk/sql/execution/datasources/parquet/ParquetFilters.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala 7 additions, 6 deletions...k/sql/execution/datasources/parquet/ParquetRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala 41 additions, 2 deletions...ql/execution/datasources/parquet/ParquetFilterSuite.scala
Loading
Please register or sign in to comment