-
- Downloads
[SPARK-6443] [SPARK SUBMIT] Could not submit app in standalone cluster mode when HA is enabled
**3/26 update:** * Akka-based: Use an array of `ActorSelection` to represent multiple master. Add an `activeMasterActor` for query status of driver. And will add lost masters( including the standby one) to `lostMasters`. When size of `lostMasters` equals or greater than # of all masters, we should give an error that all masters are not avalible. * Rest-based: When all masters are not available(throw an exception), we use akka gateway to submit apps. I have tested simply on standalone HA cluster(with two masters alive and one alive/one dead), it worked. There might remains some issues on style or message print, but we can check the solution then fix them together. /cc srowen andrewor14 Author: WangTaoTheTonic <wangtao111@huawei.com> Closes #5116 from WangTaoTheTonic/SPARK-6443 and squashes the following commits: 2a28aab [WangTaoTheTonic] based the newest change https://github.com/apache/spark/pull/5144 76fd411 [WangTaoTheTonic] rebase f4f972b [WangTaoTheTonic] rebase...again a41de0b [WangTaoTheTonic] rebase 220cb3c [WangTaoTheTonic] move connect exception inside 35119a0 [WangTaoTheTonic] style and compile issues 9d636be [WangTaoTheTonic] per Andrew's comments 979760c [WangTaoTheTonic] rebase e4f4ece [WangTaoTheTonic] fix failed test 5d23958 [WangTaoTheTonic] refact some duplicated code, style and comments 7a881b3 [WangTaoTheTonic] when one of masters is gone, we still can submit 2b011c9 [WangTaoTheTonic] fix broken tests 60d97a4 [WangTaoTheTonic] rebase fa1fa80 [WangTaoTheTonic] submit app to HA cluster in standalone cluster mode
Showing
- core/src/main/scala/org/apache/spark/deploy/Client.scala 55 additions, 18 deletionscore/src/main/scala/org/apache/spark/deploy/Client.scala
- core/src/main/scala/org/apache/spark/deploy/ClientArguments.scala 4 additions, 5 deletions.../main/scala/org/apache/spark/deploy/ClientArguments.scala
- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 4 additions, 4 deletions.../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 16 additions, 8 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala 111 additions, 41 deletions...a/org/apache/spark/deploy/rest/RestSubmissionClient.scala
- core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala 1 addition, 1 deletion...cala/org/apache/spark/deploy/worker/WorkerArguments.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 16 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala 22 additions, 18 deletions.../apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
Loading
Please register or sign in to comment