-
- Downloads
[SPARK-19411][SQL] Remove the metadata used to mark optional columns in merged...
[SPARK-19411][SQL] Remove the metadata used to mark optional columns in merged Parquet schema for filter predicate pushdown ## What changes were proposed in this pull request? There is a metadata introduced before to mark the optional columns in merged Parquet schema for filter predicate pushdown. As we upgrade to Parquet 1.8.2 which includes the fix for the pushdown of optional columns, we don't need this metadata now. ## How was this patch tested? Jenkins tests. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #16756 from viirya/remove-optional-metadata.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 2 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala 2 additions, 13 deletions...rc/main/scala/org/apache/spark/sql/types/StructType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 0 additions, 49 deletions...test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala 2 additions, 8 deletions...sql/execution/datasources/parquet/ParquetFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala 2 additions, 11 deletions...rk/sql/execution/datasources/parquet/ParquetFilters.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala 14 additions, 54 deletions...ql/execution/datasources/parquet/ParquetFilterSuite.scala
Loading
Please register or sign in to comment