-
- Downloads
[SPARK-3762] clear reference of SparkEnv after stop
SparkEnv is cached in ThreadLocal object, so after stop and create a new SparkContext, old SparkEnv is still used by some threads, it will trigger many problems, for example, pyspark will have problem after restart SparkContext, because py4j use thread pool for RPC. This patch will clear all the references after stop a SparkEnv. cc mateiz tdas pwendell Author: Davies Liu <davies.liu@gmail.com> Closes #2624 from davies/env and squashes the following commits: a69f30c [Davies Liu] deprecate getThreadLocal ba77ca4 [Davies Liu] remove getThreadLocal(), update docs ee62bb7 [Davies Liu] cleanup ThreadLocal of SparnENV 4d0ea8b [Davies Liu] clear reference of SparkEnv after stop
Showing
- core/src/main/scala/org/apache/spark/SparkEnv.scala 8 additions, 11 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 0 additions, 1 deletion...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 0 additions, 2 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala 0 additions, 1 deletioncore/src/main/scala/org/apache/spark/rdd/PipedRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 0 additions, 1 deletion.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 0 additions, 2 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala 0 additions, 1 deletion...a/org/apache/spark/streaming/scheduler/JobGenerator.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala 0 additions, 1 deletion...a/org/apache/spark/streaming/scheduler/JobScheduler.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala 0 additions, 1 deletion...rg/apache/spark/streaming/scheduler/ReceiverTracker.scala
Loading
Please register or sign in to comment