-
- Downloads
[SPARK-15355][CORE] Proactive block replication
## What changes were proposed in this pull request? We are proposing addition of pro-active block replication in case of executor failures. BlockManagerMasterEndpoint does all the book-keeping to keep a track of all the executors and the blocks they hold. It also keeps a track of which executors are alive through heartbeats. When an executor is removed, all this book-keeping state is updated to reflect the lost executor. This step can be used to identify executors that are still in possession of a copy of the cached data and a message could be sent to them to use the existing "replicate" function to find and place new replicas on other suitable hosts. Blocks replicated this way will let the master know of their existence. This can happen when an executor is lost, and would that way be pro-active as opposed be being done at query time. ## How was this patch tested? This patch was tested with existing unit tests along with new unit tests added to test the functionality. Author: Shubham Chopra <schopra31@bloomberg.net> Closes #14412 from shubhamchopra/ProactiveBlockReplication.
Showing
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 37 additions, 6 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala 24 additions, 0 deletions...org/apache/spark/storage/BlockManagerMasterEndpoint.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala 4 additions, 0 deletions...scala/org/apache/spark/storage/BlockManagerMessages.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerSlaveEndpoint.scala 4 additions, 0 deletions.../org/apache/spark/storage/BlockManagerSlaveEndpoint.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala 76 additions, 19 deletions...g/apache/spark/storage/BlockManagerReplicationSuite.scala
- docs/configuration.md 9 additions, 0 deletionsdocs/configuration.md
Loading
Please register or sign in to comment