-
- Downloads
Standalone Scheduler fault tolerance using ZooKeeper
This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch 194ba4b8), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from 194ba4b8. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again. Forthcoming: Documentation, tests (! - only ad hoc testing has been performed so far) I do not intend for this commit to be merged until tests are added, but this patch should still be mostly reviewable until then.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 4 additions, 3 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala 5 additions, 15 deletions...rc/main/scala/org/apache/spark/deploy/DeployMessage.scala
- core/src/main/scala/org/apache/spark/deploy/ExecutorDescription.scala 5 additions, 0 deletions...n/scala/org/apache/spark/deploy/ExecutorDescription.scala
- core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala 4 additions, 3 deletions...ain/scala/org/apache/spark/deploy/LocalSparkCluster.scala
- core/src/main/scala/org/apache/spark/deploy/client/Client.scala 45 additions, 13 deletions...rc/main/scala/org/apache/spark/deploy/client/Client.scala
- core/src/main/scala/org/apache/spark/deploy/client/ClientListener.scala 4 additions, 0 deletions...scala/org/apache/spark/deploy/client/ClientListener.scala
- core/src/main/scala/org/apache/spark/deploy/client/TestClient.scala 6 additions, 1 deletion...ain/scala/org/apache/spark/deploy/client/TestClient.scala
- core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala 9 additions, 9 deletions...cala/org/apache/spark/deploy/master/ApplicationInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/ExecutorInfo.scala 1 addition, 1 deletion...n/scala/org/apache/spark/deploy/master/ExecutorInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala 7 additions, 7 deletions...che/spark/deploy/master/FileSystemPersistenceEngine.scala
- core/src/main/scala/org/apache/spark/deploy/master/LeaderElectionAgent.scala 28 additions, 0 deletions.../org/apache/spark/deploy/master/LeaderElectionAgent.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 90 additions, 55 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/master/MasterMessages.scala 29 additions, 0 deletions...scala/org/apache/spark/deploy/master/MasterMessages.scala
- core/src/main/scala/org/apache/spark/deploy/master/MasterState.scala 2 additions, 2 deletions...in/scala/org/apache/spark/deploy/master/MasterState.scala
- core/src/main/scala/org/apache/spark/deploy/master/PersistenceEngine.scala 3 additions, 1 deletion...la/org/apache/spark/deploy/master/PersistenceEngine.scala
- core/src/main/scala/org/apache/spark/deploy/master/SparkZooKeeperSession.scala 183 additions, 0 deletions...rg/apache/spark/deploy/master/SparkZooKeeperSession.scala
- core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala 12 additions, 12 deletions...ain/scala/org/apache/spark/deploy/master/WorkerInfo.scala
- core/src/main/scala/org/apache/spark/deploy/master/ZooKeeperLeaderElectionAgent.scala 109 additions, 0 deletions...he/spark/deploy/master/ZooKeeperLeaderElectionAgent.scala
- core/src/main/scala/org/apache/spark/deploy/master/ZooKeeperPersistenceEngine.scala 64 additions, 0 deletions...ache/spark/deploy/master/ZooKeeperPersistenceEngine.scala
- core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala 84 additions, 41 deletions...rc/main/scala/org/apache/spark/deploy/worker/Worker.scala
Loading
Please register or sign in to comment