-
- Downloads
[SPARK-5296] [SQL] Add more filter types for data sources API
This PR adds the following filter types for data sources API: - `IsNull` - `IsNotNull` - `Not` - `And` - `Or` The code which converts Catalyst predicate expressions to data sources filters is very similar to filter conversion logics in `ParquetFilters` which converts Catalyst predicates to Parquet filter predicates. In this way we can support nested AND/OR/NOT predicates without changing current `BaseScan` type hierarchy. <!-- Reviewable:start --> [<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/4623) <!-- Reviewable:end --> Author: Cheng Lian <lian@databricks.com> This patch had conflicts when merged, resolved by Committer: Michael Armbrust <michael@databricks.com> Closes #4623 from liancheng/more-fiters and squashes the following commits: 1b296f4 [Cheng Lian] Add more filter types for data sources API
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 5 additions, 4 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala 4 additions, 1 deletion.../main/scala/org/apache/spark/sql/parquet/newParquet.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala 57 additions, 24 deletions...ala/org/apache/spark/sql/sources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/filters.scala 5 additions, 0 deletions...src/main/scala/org/apache/spark/sql/sources/filters.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala 32 additions, 2 deletions...cala/org/apache/spark/sql/sources/FilteredScanSuite.scala
Loading
Please register or sign in to comment