-
- Downloads
[SPARK-16134][SQL] optimizer rules for typed filter
## What changes were proposed in this pull request? This PR adds 3 optimizer rules for typed filter: 1. push typed filter down through `SerializeFromObject` and eliminate the deserialization in filter condition. 2. pull typed filter up through `SerializeFromObject` and eliminate the deserialization in filter condition. 3. combine adjacent typed filters and share the deserialized object among all the condition expressions. This PR also adds `TypedFilter` logical plan, to separate it from normal filter, so that the concept is more clear and it's easier to write optimizer rules. ## How was this patch tested? `TypedFilterOptimizationSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #13846 from cloud-fan/filter.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 1 addition, 5 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ReferenceToExpressions.scala 1 addition, 0 deletions...ark/sql/catalyst/expressions/ReferenceToExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 46 additions, 52 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala 43 additions, 4 deletions.../org/apache/spark/sql/catalyst/plans/logical/object.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/TypedFilterOptimizationSuite.scala 66 additions, 20 deletions...sql/catalyst/optimizer/TypedFilterOptimizationSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 2 additions, 10 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 2 additions, 0 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 1 addition, 0 deletionssql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
Loading
Please register or sign in to comment