-
- Downloads
[SPARK-2288] Hide ShuffleBlockManager behind ShuffleManager
By Hiding the shuffleblockmanager behind Shufflemanager, we decouple the shuffle data's block mapping management work from Diskblockmananger. This give a more clear interface and more easy for other shuffle manager to implement their own block management logic. the jira ticket have more details. Author: Raymond Liu <raymond.liu@intel.com> Closes #1241 from colorant/shuffle and squashes the following commits: 0e01ae3 [Raymond Liu] Move ShuffleBlockmanager behind shuffleManager
Showing
- core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala 37 additions, 44 deletions...la/org/apache/spark/shuffle/FileShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala 121 additions, 0 deletions...a/org/apache/spark/shuffle/IndexShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockManager.scala 38 additions, 0 deletions.../scala/org/apache/spark/shuffle/ShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala 7 additions, 2 deletions.../main/scala/org/apache/spark/shuffle/ShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala 15 additions, 3 deletions...la/org/apache/spark/shuffle/hash/HashShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala 5 additions, 5 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 26 additions, 23 deletions...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 8 additions, 24 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala 1 addition, 1 deletion...scala/org/apache/spark/storage/BlockFetcherIterator.scala
- core/src/main/scala/org/apache/spark/storage/BlockId.scala 8 additions, 0 deletionscore/src/main/scala/org/apache/spark/storage/BlockId.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 14 additions, 8 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerSlaveActor.scala 2 additions, 2 deletions...ala/org/apache/spark/storage/BlockManagerSlaveActor.scala
- core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala 5 additions, 29 deletions...ain/scala/org/apache/spark/storage/DiskBlockManager.scala
- core/src/main/scala/org/apache/spark/storage/DiskStore.scala 25 additions, 16 deletionscore/src/main/scala/org/apache/spark/storage/DiskStore.scala
- core/src/main/scala/org/apache/spark/storage/TachyonBlockManager.scala 2 additions, 2 deletions.../scala/org/apache/spark/storage/TachyonBlockManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 8 additions, 30 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/shuffle/hash/HashShuffleManagerSuite.scala 111 additions, 0 deletions...g/apache/spark/shuffle/hash/HashShuffleManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockFetcherIteratorSuite.scala 18 additions, 18 deletions.../org/apache/spark/storage/BlockFetcherIteratorSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 3 additions, 4 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/DiskBlockManagerSuite.scala 4 additions, 139 deletions...cala/org/apache/spark/storage/DiskBlockManagerSuite.scala
Loading
Please register or sign in to comment