-
- Downloads
Merge pull request #19 from aarondav/master-zk
Standalone Scheduler fault tolerance using ZooKeeper This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch d5a96fec), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from d5a96fec. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again.
No related branches found
No related tags found
Showing
- bin/stop-slaves.sh 0 additions, 2 deletionsbin/stop-slaves.sh
- core/src/main/scala/org/apache/spark/SparkContext.scala 5 additions, 4 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala 18 additions, 11 deletions...rc/main/scala/org/apache/spark/deploy/DeployMessage.scala
- core/src/main/scala/org/apache/spark/deploy/ExecutorDescription.scala 34 additions, 0 deletions...n/scala/org/apache/spark/deploy/ExecutorDescription.scala
- core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala 420 additions, 0 deletions...in/scala/org/apache/spark/deploy/FaultToleranceTest.scala
- core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala 2 additions, 1 deletion...src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
- core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala 4 additions, 3 deletions...ain/scala/org/apache/spark/deploy/LocalSparkCluster.scala
- core/src/main/scala/org/apache/spark/deploy/client/Client.scala 68 additions, 16 deletions...rc/main/scala/org/apache/spark/deploy/client/Client.scala
- core/src/main/scala/org/apache/spark/deploy/client/ClientListener.scala 4 additions, 0 deletions...scala/org/apache/spark/deploy/client/ClientListener.scala
- core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala 6 additions, 1 deletion...ain/scala/org/apache/spark/deploy/client/TestClient.scala
- core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala 38 additions, 15 deletions...cala/org/apache/spark/deploy/master/ApplicationInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/ApplicationState.scala 2 additions, 2 deletions...ala/org/apache/spark/deploy/master/ApplicationState.scala
- core/src/main/scala/org/apache/spark/deploy/master/ExecutorInfo.scala 6 additions, 1 deletion...n/scala/org/apache/spark/deploy/master/ExecutorInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala 90 additions, 0 deletions...che/spark/deploy/master/FileSystemPersistenceEngine.scala
- core/src/main/scala/org/apache/spark/deploy/master/LeaderElectionAgent.scala 45 additions, 0 deletions.../org/apache/spark/deploy/master/LeaderElectionAgent.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 196 additions, 32 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/master/MasterMessages.scala 46 additions, 0 deletions...scala/org/apache/spark/deploy/master/MasterMessages.scala
- core/src/main/scala/org/apache/spark/deploy/master/PersistenceEngine.scala 53 additions, 0 deletions...la/org/apache/spark/deploy/master/PersistenceEngine.scala
- core/src/main/scala/org/apache/spark/deploy/master/RecoveryState.scala 26 additions, 0 deletions.../scala/org/apache/spark/deploy/master/RecoveryState.scala
- core/src/main/scala/org/apache/spark/deploy/master/SparkZooKeeperSession.scala 203 additions, 0 deletions...rg/apache/spark/deploy/master/SparkZooKeeperSession.scala
Loading
Please register or sign in to comment