-
- Downloads
[SPARK-10582][YARN][CORE] Fix AM failure situation for dynamic allocation
Because of AM failure, the target executor number between driver and AM will be different, which will lead to unexpected behavior in dynamic allocation. So when AM is re-registered with driver, state in `ExecutorAllocationManager` and `CoarseGrainedSchedulerBacked` should be reset. This issue is originally addressed in #8737 , here re-opened again. Thanks a lot KaiXinXiaoLei for finding this issue. andrewor14 and vanzin would you please help to review this, thanks a lot. Author: jerryshao <sshao@hortonworks.com> Closes #9963 from jerryshao/SPARK-10582.
Showing
- core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala 16 additions, 2 deletions...in/scala/org/apache/spark/ExecutorAllocationManager.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 19 additions, 0 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala 84 additions, 0 deletions...ala/org/apache/spark/ExecutorAllocationManagerSuite.scala
- yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala 23 additions, 0 deletions...apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
Loading
Please register or sign in to comment