Skip to content
Snippets Groups Projects
Commit d28c6754 authored by Herman van Hovell's avatar Herman van Hovell Committed by Yin Huai
Browse files

[SPARK-14986][SQL] Return correct result for empty LATERAL VIEW OUTER

## What changes were proposed in this pull request?
A Generate with the `outer` flag enabled should always return one or more rows for every input row. The optimizer currently violates this by rewriting `outer` Generates that do not contain columns of the child plan into an unjoined generate, for example:
```sql
select e from a lateral view outer explode(a.b) as e
```
The result of this is that `outer` Generate does not produce output at all when the Generators' input expression is empty. This PR fixes this.

## How was this patch tested?
Added test case to `SQLQuerySuite`.

Author: Herman van Hovell <hvanhovell@questtec.nl>

Closes #12906 from hvanhovell/SPARK-14986.
parent 89f73f67
No related branches found
No related tags found
No related merge requests found
...@@ -362,7 +362,8 @@ object ColumnPruning extends Rule[LogicalPlan] { ...@@ -362,7 +362,8 @@ object ColumnPruning extends Rule[LogicalPlan] {
g.copy(child = prunedChild(g.child, g.references)) g.copy(child = prunedChild(g.child, g.references))
// Turn off `join` for Generate if no column from it's child is used // Turn off `join` for Generate if no column from it's child is used
case p @ Project(_, g: Generate) if g.join && p.references.subsetOf(g.generatedSet) => case p @ Project(_, g: Generate)
if g.join && !g.outer && p.references.subsetOf(g.generatedSet) =>
p.copy(child = g.copy(join = false)) p.copy(child = g.copy(join = false))
// Eliminate unneeded attributes from right side of a Left Existence Join. // Eliminate unneeded attributes from right side of a Left Existence Join.
......
...@@ -2473,4 +2473,11 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext { ...@@ -2473,4 +2473,11 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
Row("r3c1x", "r3c2", "t1r3c3", "r3c2", "t1r3c3") :: Nil) Row("r3c1x", "r3c2", "t1r3c3", "r3c2", "t1r3c3") :: Nil)
} }
} }
test("SPARK-14986: Outer lateral view with empty generate expression") {
checkAnswer(
sql("select nil from (select 1 as x ) x lateral view outer explode(array()) n as nil"),
Row(null) :: Nil
)
}
} }
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment