-
- Downloads
[SPARK-18751][CORE] Fix deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext
## What changes were proposed in this pull request? When `SparkContext.stop` is called in `Utils.tryOrStopSparkContext` (the following three places), it will cause deadlock because the `stop` method needs to wait for the thread running `stop` to exit. - ContextCleaner.keepCleaning - LiveListenerBus.listenerThread.run - TaskSchedulerImpl.start This PR adds `SparkContext.stopInNewThread` and uses it to eliminate the potential deadlock. I also removed my changes in #15775 since they are not necessary now. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #16178 from zsxwing/fix-stop-deadlock. (cherry picked from commit 26432df9) Signed-off-by:Shixiong Zhu <shixiong@databricks.com>
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 20 additions, 15 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala 0 additions, 5 deletionscore/src/main/scala/org/apache/spark/rpc/RpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala 0 additions, 1 deletion...rc/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 0 additions, 5 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala 1 addition, 1 deletion.../spark/scheduler/cluster/StandaloneSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala 0 additions, 13 deletionscore/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
Loading
Please register or sign in to comment