Skip to content
Snippets Groups Projects
Commit ac27557e authored by Sameer Agarwal's avatar Sameer Agarwal Committed by Reynold Xin
Browse files

[SPARK-17228][SQL] Not infer/propagate non-deterministic constraints

## What changes were proposed in this pull request?

Given that filters based on non-deterministic constraints shouldn't be pushed down in the query plan, unnecessarily inferring them is confusing and a source of potential bugs. This patch simplifies the inferring logic by simply ignoring them.

## How was this patch tested?

Added a new test in `ConstraintPropagationSuite`.

Author: Sameer Agarwal <sameerag@cs.berkeley.edu>

Closes #14795 from sameeragarwal/deterministic-constraints.
parent 3a60be4b
No related branches found
No related tags found
No related merge requests found
......@@ -35,7 +35,8 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]] extends TreeNode[PlanT
.union(inferAdditionalConstraints(constraints))
.union(constructIsNotNullConstraints(constraints))
.filter(constraint =>
constraint.references.nonEmpty && constraint.references.subsetOf(outputSet))
constraint.references.nonEmpty && constraint.references.subsetOf(outputSet) &&
constraint.deterministic)
}
/**
......
......@@ -352,4 +352,21 @@ class ConstraintPropagationSuite extends SparkFunSuite {
verifyConstraints(tr.analyze.constraints,
ExpressionSet(Seq(IsNotNull(resolveColumn(tr, "b")), IsNotNull(resolveColumn(tr, "c")))))
}
test("not infer non-deterministic constraints") {
val tr = LocalRelation('a.int, 'b.string, 'c.int)
verifyConstraints(tr
.where('a.attr === Rand(0))
.analyze.constraints,
ExpressionSet(Seq(IsNotNull(resolveColumn(tr, "a")))))
verifyConstraints(tr
.where('a.attr === InputFileName())
.where('a.attr =!= 'c.attr)
.analyze.constraints,
ExpressionSet(Seq(resolveColumn(tr, "a") =!= resolveColumn(tr, "c"),
IsNotNull(resolveColumn(tr, "a")),
IsNotNull(resolveColumn(tr, "c")))))
}
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment