-
- Downloads
[SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL
## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/11410, we missed a corner case: define the inner class and use it in `Dataset` at the same time by using paste mode. For this case, the inner class and the `Dataset` are inside same line object, when we build the `Dataset`, we try to get outer pointer from line object, and it will fail because the line object is not initialized yet. https://issues.apache.org/jira/browse/SPARK-13456?focusedCommentId=15209174&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209174 is an example for this corner case. This PR make the process of getting outer pointer from line object lazy, so that we can successfully build the `Dataset` and finish initializing the line object. ## How was this patch tested? new test in repl suite. Author: Wenchen Fan <wenchen@databricks.com> Closes #11931 from cloud-fan/repl.
Showing
- repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala 15 additions, 0 deletions...2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala 25 additions, 14 deletions.../org/apache/spark/sql/catalyst/encoders/OuterScopes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala 7 additions, 5 deletions...a/org/apache/spark/sql/catalyst/expressions/objects.scala
Loading
Please register or sign in to comment