-
- Downloads
[SPARK-10548] [SPARK-10563] [SQL] Fix concurrent SQL executions
*Note: this is for master branch only.* The fix for branch-1.5 is at #8721. The query execution ID is currently passed from a thread to its children, which is not the intended behavior. This led to `IllegalArgumentException: spark.sql.execution.id is already set` when running queries in parallel, e.g.: ``` (1 to 100).par.foreach { _ => sc.parallelize(1 to 5).map { i => (i, i) }.toDF("a", "b").count() } ``` The cause is `SparkContext`'s local properties are inherited by default. This patch adds a way to exclude keys we don't want to be inherited, and makes SQL go through that code path. Author: Andrew Or <andrew@databricks.com> Closes #8710 from andrewor14/concurrent-sql-executions.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 7 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/test/scala/org/apache/spark/ThreadingSuite.scala 24 additions, 41 deletionscore/src/test/scala/org/apache/spark/ThreadingSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala 101 additions, 0 deletions...la/org/apache/spark/sql/execution/SQLExecutionSuite.scala
Loading
Please register or sign in to comment