-
- Downloads
[SPARK-5278][SQL] Introduce UnresolvedGetField and complete the check of...
[SPARK-5278][SQL] Introduce UnresolvedGetField and complete the check of ambiguous reference to fields When the `GetField` chain(`a.b.c.d.....`) is interrupted by `GetItem` like `a.b[0].c.d....`, then the check of ambiguous reference to fields is broken. The reason is that: for something like `a.b[0].c.d`, we first parse it to `GetField(GetField(GetItem(Unresolved("a.b"), 0), "c"), "d")`. Then in `LogicalPlan#resolve`, we resolve `"a.b"` and build a `GetField` chain from bottom(the relation). But for the 2 outer `GetFiled`, we have to resolve them in `Analyzer` or do it in `GetField` lazily, check data type of child, search needed field, etc. which is similar to what we have done in `LogicalPlan#resolve`. So in this PR, the fix is just copy the same logic in `LogicalPlan#resolve` to `Analyzer`, which is simple and quick, but I do suggest introduce `UnresolvedGetFiled` like I explained in https://github.com/apache/spark/pull/2405. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #4068 from cloud-fan/simple and squashes the following commits: a6857b5 [Wenchen Fan] fix import order 8411c40 [Wenchen Fan] use UnresolvedGetField
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 24 additions, 10 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala 12 additions, 0 deletions...a/org/apache/spark/sql/catalyst/analysis/unresolved.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 2 additions, 2 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala 2 additions, 22 deletions.../apache/spark/sql/catalyst/expressions/complexTypes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 1 addition, 1 deletion...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 2 additions, 31 deletions...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala 18 additions, 7 deletions.../sql/catalyst/expressions/ExpressionEvaluationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala 2 additions, 2 deletions...e/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 2 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 1 addition, 1 deletion...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala 17 additions, 10 deletions...apache/spark/sql/hive/execution/HiveResolutionSuite.scala
Loading
Please register or sign in to comment