Skip to content
Snippets Groups Projects
Commit a66142de authored by Josh Rosen's avatar Josh Rosen
Browse files

[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites

This patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.

This configuration leak was inadvertently introduced during test cleanup in #9618.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.
parent 3e1d120c
No related branches found
No related tags found
No related merge requests found
...@@ -868,29 +868,27 @@ class TungstenAggregationQueryWithControlledFallbackSuite extends AggregationQue ...@@ -868,29 +868,27 @@ class TungstenAggregationQueryWithControlledFallbackSuite extends AggregationQue
override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = { override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = {
(0 to 2).foreach { fallbackStartsAt => (0 to 2).foreach { fallbackStartsAt =>
sqlContext.setConf( withSQLConf("spark.sql.TungstenAggregate.testFallbackStartsAt" -> fallbackStartsAt.toString) {
"spark.sql.TungstenAggregate.testFallbackStartsAt", // Create a new df to make sure its physical operator picks up
fallbackStartsAt.toString) // spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it?
// Create a new df to make sure its physical operator picks up val newActual = DataFrame(sqlContext, actual.logicalPlan)
// spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it? QueryTest.checkAnswer(newActual, expectedAnswer) match {
val newActual = DataFrame(sqlContext, actual.logicalPlan) case Some(errorMessage) =>
val newErrorMessage =
QueryTest.checkAnswer(newActual, expectedAnswer) match { s"""
case Some(errorMessage) => |The following aggregation query failed when using TungstenAggregate with
val newErrorMessage = |controlled fallback (it falls back to sort-based aggregation once it has processed
s""" |$fallbackStartsAt input rows). The query is
|The following aggregation query failed when using TungstenAggregate with |${actual.queryExecution}
|controlled fallback (it falls back to sort-based aggregation once it has processed |
|$fallbackStartsAt input rows). The query is |$errorMessage
|${actual.queryExecution} """.stripMargin
|
|$errorMessage fail(newErrorMessage)
""".stripMargin case None =>
}
fail(newErrorMessage)
case None =>
} }
} }
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment