-
- Downloads
[SPARK-15353][CORE] Making peer selection for block replication pluggable
## What changes were proposed in this pull request? This PR makes block replication strategies pluggable. It provides two trait that can be implemented, one that maps a host to its topology and is used in the master, and the second that helps prioritize a list of peers for block replication and would run in the executors. This patch contains default implementations of these traits that make sure current Spark behavior is unchanged. ## How was this patch tested? This patch should not change Spark behavior in any way, and was tested with unit tests for storage. Author: Shubham Chopra <schopra31@bloomberg.net> Closes #13152 from shubhamchopra/RackAwareBlockReplication.
Showing
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 82 additions, 85 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala 27 additions, 7 deletions.../main/scala/org/apache/spark/storage/BlockManagerId.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 12 additions, 4 deletions...n/scala/org/apache/spark/storage/BlockManagerMaster.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala 29 additions, 3 deletions...org/apache/spark/storage/BlockManagerMasterEndpoint.scala
- core/src/main/scala/org/apache/spark/storage/BlockReplicationPolicy.scala 112 additions, 0 deletions...ala/org/apache/spark/storage/BlockReplicationPolicy.scala
- core/src/main/scala/org/apache/spark/storage/TopologyMapper.scala 86 additions, 0 deletions.../main/scala/org/apache/spark/storage/TopologyMapper.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala 2 additions, 0 deletions...g/apache/spark/storage/BlockManagerReplicationSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala 74 additions, 0 deletions...rg/apache/spark/storage/BlockReplicationPolicySuite.scala
- core/src/test/scala/org/apache/spark/storage/TopologyMapperSuite.scala 68 additions, 0 deletions.../scala/org/apache/spark/storage/TopologyMapperSuite.scala
Loading
Please register or sign in to comment