-
- Downloads
[SPARK-12718][SPARK-13720][SQL] SQL generation support for window functions
## What changes were proposed in this pull request? Add SQL generation support for window functions. The idea is simple, just treat `Window` operator like `Project`, i.e. add subquery to its child when necessary, generate a `SELECT ... FROM ...` SQL string, implement `sql` method for window related expressions, e.g. `WindowSpecDefinition`, `WindowFrame`, etc. This PR also fixed SPARK-13720 by improving the process of adding extra `SubqueryAlias`(the `RecoverScopingInfo` rule). Before this PR, we update the qualifiers in project list while adding the subquery. However, this is incomplete as we need to update qualifiers in all ancestors that refer attributes here. In this PR, we split `RecoverScopingInfo` into 2 rules: `AddSubQuery` and `UpdateQualifier`. `AddSubQuery` only add subquery if necessary, and `UpdateQualifier` will re-propagate and update qualifiers bottom up. Ideally we should put the bug fix part in an individual PR, but this bug also blocks the window stuff, so I put them together here. Many thanks to gatorsmile for the initial discussion and test cases! ## How was this patch tested? new tests in `LogicalPlanToSQLSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #11555 from cloud-fan/window.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 3 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala 2 additions, 1 deletion...org/apache/spark/sql/catalyst/expressions/SortOrder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala 42 additions, 5 deletions...he/spark/sql/catalyst/expressions/windowExpressions.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/SQLBuilder.scala 77 additions, 27 deletions...src/main/scala/org/apache/spark/sql/hive/SQLBuilder.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala 111 additions, 3 deletions...ala/org/apache/spark/sql/hive/LogicalPlanToSQLSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala 1 addition, 0 deletions.../apache/spark/sql/hive/execution/HiveComparisonTest.scala
Loading
Please register or sign in to comment