-
- Downloads
[SPARK-17365][CORE] Remove/Kill multiple executors together to reduce RPC call time.
## What changes were proposed in this pull request? We are killing multiple executors together instead of iterating over expensive RPC calls to kill single executor. ## How was this patch tested? Executed sample spark job to observe executors being killed/removed with dynamic allocation enabled. Author: Dhruve Ashar <dashar@yahoo-inc.com> Author: Dhruve Ashar <dhruveashar@gmail.com> Closes #15152 from dhruve/impr/SPARK-17365.
Showing
- core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala 6 additions, 3 deletions...ain/scala/org/apache/spark/ExecutorAllocationClient.scala
- core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala 64 additions, 22 deletions...in/scala/org/apache/spark/ExecutorAllocationManager.scala
- core/src/main/scala/org/apache/spark/SparkContext.scala 15 additions, 9 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 9 additions, 3 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/test/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager 2 additions, 1 deletion...ervices/org.apache.spark.scheduler.ExternalClusterManager
- core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala 125 additions, 10 deletions...ala/org/apache/spark/ExecutorAllocationManagerSuite.scala
- core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala 3 additions, 3 deletions...pache/spark/deploy/StandaloneDynamicAllocationSuite.scala
- project/MimaExcludes.scala 3 additions, 0 deletionsproject/MimaExcludes.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala 1 addition, 1 deletion...spark/streaming/scheduler/ExecutorAllocationManager.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala 8 additions, 1 deletion...a/org/apache/spark/streaming/scheduler/JobScheduler.scala
- streaming/src/test/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManagerSuite.scala 3 additions, 2 deletions.../streaming/scheduler/ExecutorAllocationManagerSuite.scala
Loading
Please register or sign in to comment