-
- Downloads
[SPARK-13721][SQL] Support outer generators in DataFrame API
## What changes were proposed in this pull request? Added outer_explode, outer_posexplode, outer_inline functions and expressions. Some bug fixing in GenerateExec.scala for CollectionGenerator. Previously it was not correctly handling the case of outer with empty collections, only with nulls. ## How was this patch tested? New tests added to GeneratorFunctionSuite Author: Bogdan Raducanu <bogdan.rdc@gmail.com> Closes #16608 from bogdanrdc/SPARK-13721.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 14 additions, 6 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 12 additions, 0 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala 9 additions, 0 deletions...rg/apache/spark/sql/catalyst/expressions/generators.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 11 additions, 4 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 1 addition, 4 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala 6 additions, 2 deletions...n/scala/org/apache/spark/sql/execution/GenerateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 18 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala 76 additions, 0 deletions...t/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/catalyst/ExpressionToSQLSuite.scala 3 additions, 0 deletions.../org/apache/spark/sql/catalyst/ExpressionToSQLSuite.scala
Loading
Please register or sign in to comment