-
- Downloads
[SPARK-4391][SQL] Configure parquet filters using SQLConf
This is more uniform with the rest of SQL configuration and allows it to be turned on and off without restarting the SparkContext. In this PR I also turn off filter pushdown by default due to a number of outstanding issues (in particular SPARK-4258). When those are fixed we should turn it back on by default. Author: Michael Armbrust <michael@databricks.com> Closes #3258 from marmbrus/parquetFilters and squashes the following commits: 5655bfe [Michael Armbrust] Remove extra line. 15e9a98 [Michael Armbrust] Enable filters for tests 75afd39 [Michael Armbrust] Fix comments 78fa02d [Michael Armbrust] off by default e7f9e16 [Michael Armbrust] First draft of correctly configuring parquet filter pushdown
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 7 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 5 additions, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetFilters.scala 0 additions, 2 deletions...n/scala/org/apache/spark/sql/parquet/ParquetFilters.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala 7 additions, 6 deletions...org/apache/spark/sql/parquet/ParquetTableOperations.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala 2 additions, 0 deletions...cala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala
Loading
Please register or sign in to comment