Skip to content
Snippets Groups Projects
Commit a66142de authored by Josh Rosen's avatar Josh Rosen
Browse files

[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites

This patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.

This configuration leak was inadvertently introduced during test cleanup in #9618.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.
parent 3e1d120c
No related branches found
No related tags found
No related merge requests found
......@@ -868,29 +868,27 @@ class TungstenAggregationQueryWithControlledFallbackSuite extends AggregationQue
override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = {
(0 to 2).foreach { fallbackStartsAt =>
sqlContext.setConf(
"spark.sql.TungstenAggregate.testFallbackStartsAt",
fallbackStartsAt.toString)
// Create a new df to make sure its physical operator picks up
// spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it?
val newActual = DataFrame(sqlContext, actual.logicalPlan)
QueryTest.checkAnswer(newActual, expectedAnswer) match {
case Some(errorMessage) =>
val newErrorMessage =
s"""
|The following aggregation query failed when using TungstenAggregate with
|controlled fallback (it falls back to sort-based aggregation once it has processed
|$fallbackStartsAt input rows). The query is
|${actual.queryExecution}
|
|$errorMessage
""".stripMargin
fail(newErrorMessage)
case None =>
withSQLConf("spark.sql.TungstenAggregate.testFallbackStartsAt" -> fallbackStartsAt.toString) {
// Create a new df to make sure its physical operator picks up
// spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it?
val newActual = DataFrame(sqlContext, actual.logicalPlan)
QueryTest.checkAnswer(newActual, expectedAnswer) match {
case Some(errorMessage) =>
val newErrorMessage =
s"""
|The following aggregation query failed when using TungstenAggregate with
|controlled fallback (it falls back to sort-based aggregation once it has processed
|$fallbackStartsAt input rows). The query is
|${actual.queryExecution}
|
|$errorMessage
""".stripMargin
fail(newErrorMessage)
case None =>
}
}
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment