-
- Downloads
[SPARK-16958] [SQL] Reuse subqueries within the same query
## What changes were proposed in this pull request? There could be multiple subqueries that generate same results, we could re-use the result instead of running it multiple times. This PR also cleanup up how we run subqueries. For SQL query ```sql select id,(select avg(id) from t) from t where id > (select avg(id) from t) ``` The explain is ``` == Physical Plan == *Project [id#15L, Subquery subquery29 AS scalarsubquery()#35] : +- Subquery subquery29 : +- *HashAggregate(keys=[], functions=[avg(id#15L)]) : +- Exchange SinglePartition : +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)]) : +- *Range (0, 1000, splits=4) +- *Filter (cast(id#15L as double) > Subquery subquery29) : +- Subquery subquery29 : +- *HashAggregate(keys=[], functions=[avg(id#15L)]) : +- Exchange SinglePartition : +- *HashAggregate(keys=[], functions=[partial_avg(id#15L)]) : +- *Range (0, 1000, splits=4) +- *Range (0, 1000, splits=4) ``` The visualized plan:  ## How was this patch tested? Existing tests. Author: Davies Liu <davies@databricks.com> Closes #14548 from davies/subq.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala 7 additions, 0 deletions.../org/apache/spark/sql/catalyst/expressions/subquery.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 2 additions, 2 deletions.../scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 2 additions, 1 deletion...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 10 additions, 24 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala 59 additions, 4 deletions...g/apache/spark/sql/execution/basicPhysicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala 129 additions, 16 deletions.../main/scala/org/apache/spark/sql/execution/subquery.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala 6 additions, 2 deletions...la/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala
Loading
Please register or sign in to comment