-
- Downloads
[SPARK-16839][SQL] Simplify Struct creation code path
## What changes were proposed in this pull request? Simplify struct creation, especially the aspect of `CleanupAliases` which missed some aliases when handling trees created by `CreateStruct`. This PR includes: 1. A failing test (create struct with nested aliases, some of the aliases survive `CleanupAliases`). 2. A fix that transforms `CreateStruct` into a `CreateNamedStruct` constructor, effectively eliminating `CreateStruct` from all expression trees. 3. A `NamePlaceHolder` used by `CreateStruct` when column names cannot be extracted from unresolved `NamedExpression`. 4. A new Analyzer rule that resolves `NamePlaceHolder` into a string literal once the `NamedExpression` is resolved. 5. `CleanupAliases` code was simplified as it no longer has to deal with `CreateStruct`'s top level columns. ## How was this patch tested? Running all tests-suits in package org.apache.spark.sql, especially including the analysis suite, making sure added test initially fails, after applying suggested fix rerun the entire analysis package successfully. Modified few tests that expected `CreateStruct` which is now transformed into `CreateNamedStruct`. Author: eyal farago <eyal farago> Author: Herman van Hovell <hvanhovell@databricks.com> Author: eyal farago <eyal.farago@gmail.com> Author: Eyal Farago <eyal.farago@actimize.com> Author: Hyukjin Kwon <gurwls223@gmail.com> Author: eyalfa <eyal.farago@gmail.com> Closes #15718 from hvanhovell/SPARK-16839-2.
Showing
- R/pkg/inst/tests/testthat/test_sparkSQL.R 6 additions, 6 deletionsR/pkg/inst/tests/testthat/test_sparkSQL.R
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 27 additions, 26 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 1 addition, 1 deletion...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala 0 additions, 2 deletions...rg/apache/spark/sql/catalyst/expressions/Projection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala 71 additions, 141 deletions...e/spark/sql/catalyst/expressions/complexTypeCreator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 2 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala 34 additions, 4 deletions...rg/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala 0 additions, 1 deletion...che/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 3 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala 2 additions, 2 deletions...he/spark/sql/execution/command/AnalyzeColumnCommand.scala
- sql/core/src/test/resources/sql-tests/results/group-by.sql.out 1 addition, 1 deletion...ore/src/test/resources/sql-tests/results/group-by.sql.out
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 13 additions, 7 deletions.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- sql/hive/src/test/resources/sqlgen/subquery_in_having_2.sql 1 addition, 1 deletionsql/hive/src/test/resources/sqlgen/subquery_in_having_2.sql
- sql/hive/src/test/scala/org/apache/spark/sql/catalyst/LogicalPlanToSQLSuite.scala 8 additions, 4 deletions...org/apache/spark/sql/catalyst/LogicalPlanToSQLSuite.scala
Loading
Please register or sign in to comment