-
- Downloads
Never store shuffle blocks in BlockManager
After the BlockId refactor (PR #114), it became very clear that ShuffleBlocks are of no use within BlockManager (they had a no-arg constructor!). This patch completely eliminates them, saving us around 100-150 bytes per shuffle block. The total, system-wide overhead per shuffle block is now a flat 8 bytes, excluding state saved by the MapOutputTracker.
Showing
- core/src/main/scala/org/apache/spark/storage/BlockInfo.scala 1 addition, 17 deletionscore/src/main/scala/org/apache/spark/storage/BlockInfo.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 2 additions, 8 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala 1 addition, 11 deletions...in/scala/org/apache/spark/storage/BlockObjectWriter.scala
Loading
Please register or sign in to comment