-
- Downloads
[SPARK-6376][SQL] Avoid eliminating subqueries until optimization
Previously it was okay to throw away subqueries after analysis, as we would never try to use that tree for resolution again. However, with eager analysis in `DataFrame`s this can cause errors for queries such as: ```scala val df = Seq(1,2,3).map(i => (i, i.toString)).toDF("int", "str") df.as('x).join(df.as('y), $"x.str" === $"y.str").groupBy("x.str").count() ``` As a result, in this PR we defer the elimination of subqueries until the optimization phase. Author: Michael Armbrust <michael@databricks.com> Closes #5160 from marmbrus/subqueriesInDfs and squashes the following commits: a9bb262 [Michael Armbrust] Update Optimizer.scala 27d25bf [Michael Armbrust] fix hive tests 9137e03 [Michael Armbrust] add type 81cd597 [Michael Armbrust] Avoid eliminating subqueries until optimization
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 1 addition, 3 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 4 additions, 0 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 10 additions, 6 deletions...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala 6 additions, 2 deletions...rg/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 7 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala 2 additions, 2 deletionssql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala 1 addition, 1 deletion...org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala 1 addition, 1 deletion.../test/scala/org/apache/spark/sql/hive/parquetSuites.scala
Loading
Please register or sign in to comment