Skip to content
Snippets Groups Projects
Commit df89f1d4 authored by Herman van Hovell's avatar Herman van Hovell Committed by Davies Liu
Browse files

[SPARK-15122] [SQL] Fix TPC-DS 41 - Normalize predicates before pulling them out

## What changes were proposed in this pull request?
The official TPC-DS 41 query currently fails because it contains a scalar subquery with a disjunctive correlated predicate (the correlated predicates were nested in ORs). This makes the `Analyzer` pull out the entire predicate which is wrong and causes the following (correct) analysis exception: `The correlated scalar subquery can only contain equality predicates`

This PR fixes this by first simplifing (or normalizing) the correlated predicates before pulling them out of the subquery.

## How was this patch tested?
Manual testing on TPC-DS 41, and added a test to SubquerySuite.

Author: Herman van Hovell <hvanhovell@questtec.nl>

Closes #12954 from hvanhovell/SPARK-15122.
parent 607a27a0
No related branches found
No related tags found
No related merge requests found
......@@ -26,6 +26,7 @@ import org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.encoders.OuterScopes
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.optimizer.BooleanSimplification
import org.apache.spark.sql.catalyst.planning.IntegerIndex
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, _}
......@@ -958,7 +959,8 @@ class Analyzer(
localPredicateReferences -- p.outputSet
}
val transformed = sub transformUp {
// Simplify the predicates before pulling them out.
val transformed = BooleanSimplification(sub) transformUp {
case f @ Filter(cond, child) =>
// Find all predicates with an outer reference.
val (correlated, local) = splitConjunctivePredicates(cond).partition(containsOuter)
......
......@@ -281,4 +281,16 @@ class SubquerySuite extends QueryTest with SharedSQLContext {
assert(msg1.getMessage.contains(
"The correlated scalar subquery can only contain equality predicates"))
}
test("disjunctive correlated scalar subquery") {
checkAnswer(
sql("""
|select a
|from l
|where (select count(*)
| from r
| where (a = c and d = 2.0) or (a = c and d = 1.0)) > 0
""".stripMargin),
Row(3) :: Nil)
}
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment