Skip to content
Snippets Groups Projects
Commit b6071a70 authored by gatorsmile's avatar gatorsmile Committed by Michael Armbrust
Browse files

[SPARK-13722][SQL] No Push Down for Non-deterministics Predicates through Generate

#### What changes were proposed in this pull request?

Non-deterministic predicates should not be pushed through Generate.

#### How was this patch tested?

Added a test case in `FilterPushdownSuite.scala`

Author: gatorsmile <gatorsmile@gmail.com>

Closes #11562 from gatorsmile/pushPredicateDownWindow.
parent a3ec50a4
No related branches found
No related tags found
No related merge requests found
......@@ -901,7 +901,7 @@ object PushPredicateThroughGenerate extends Rule[LogicalPlan] with PredicateHelp
// Predicates that reference attributes produced by the `Generate` operator cannot
// be pushed below the operator.
val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition { cond =>
cond.references subsetOf g.child.outputSet
cond.references.subsetOf(g.child.outputSet) && cond.deterministic
}
if (pushDown.nonEmpty) {
val pushDownPredicate = pushDown.reduce(And)
......
......@@ -496,6 +496,24 @@ class FilterPushdownSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
test("generate: non-deterministic predicate referenced no generated column") {
val originalQuery = {
testRelationWithArrayType
.generate(Explode('c_arr), true, false, Some("arr"))
.where(('b >= 5) && ('a + Rand(10).as("rnd") > 6))
}
val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer = {
testRelationWithArrayType
.where('b >= 5)
.generate(Explode('c_arr), true, false, Some("arr"))
.where('a + Rand(10).as("rnd") > 6)
.analyze
}
comparePlans(optimized, correctAnswer)
}
test("generate: part of conjuncts referenced generated column") {
val generator = Explode('c_arr)
val originalQuery = {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment