-
- Downloads
[SPARK-14135] Add off-heap storage memory bookkeeping support to MemoryManager
This patch extends Spark's `UnifiedMemoryManager` to add bookkeeping support for off-heap storage memory, an requirement for enabling off-heap caching (which will be done by #11805). The `MemoryManager`'s `storageMemoryPool` has been split into separate on- and off-heap pools and the storage and unroll memory allocation methods have been updated to accept a `memoryMode` parameter to specify whether allocations should be performed on- or off-heap. In order to reduce the testing surface, the `StaticMemoryManager` does not support off-heap caching (we plan to eventually remove the `StaticMemoryManager`, so this isn't a significant limitation). Author: Josh Rosen <joshrosen@databricks.com> Closes #11942 from JoshRosen/off-heap-storage-memory-bookkeeping.
Showing
- core/src/main/scala/org/apache/spark/memory/ExecutionMemoryPool.scala 7 additions, 2 deletions...n/scala/org/apache/spark/memory/ExecutionMemoryPool.scala
- core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 30 additions, 16 deletions...rc/main/scala/org/apache/spark/memory/MemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/StaticMemoryManager.scala 25 additions, 11 deletions...n/scala/org/apache/spark/memory/StaticMemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala 15 additions, 3 deletions...ain/scala/org/apache/spark/memory/StorageMemoryPool.scala
- core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala 97 additions, 71 deletions.../scala/org/apache/spark/memory/UnifiedMemoryManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/StorageLevel.scala 6 additions, 0 deletions...rc/main/scala/org/apache/spark/storage/StorageLevel.scala
- core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala 10 additions, 9 deletions...n/scala/org/apache/spark/storage/memory/MemoryStore.scala
- core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala 1 addition, 1 deletion...st/scala/org/apache/spark/memory/MemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala 32 additions, 28 deletions...la/org/apache/spark/memory/StaticMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala 10 additions, 4 deletions...est/scala/org/apache/spark/memory/TestMemoryManager.scala
- core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala 43 additions, 37 deletions...a/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 2 additions, 2 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
Loading
Please register or sign in to comment