-
- Downloads
[SPARK-10956] Common MemoryManager interface for storage and execution
This patch introduces a `MemoryManager` that is the central arbiter of how much memory to grant to storage and execution. This patch is primarily concerned only with refactoring while preserving the existing behavior as much as possible. This is the first step away from the existing rigid separation of storage and execution memory, which has several major drawbacks discussed on the [issue](https://issues.apache.org/jira/browse/SPARK-10956). It is the precursor of a series of patches that will attempt to address those drawbacks. Author: Andrew Or <andrew@databricks.com> Author: Josh Rosen <joshrosen@databricks.com> Author: andrewor14 <andrew@databricks.com> Closes #9000 from andrewor14/memory-manager.
Showing
- core/src/main/scala/org/apache/spark/SparkEnv.scala 8 additions, 3 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 117 additions, 0 deletions...rc/main/scala/org/apache/spark/memory/MemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/StaticMemoryManager.scala 202 additions, 0 deletions...n/scala/org/apache/spark/memory/StaticMemoryManager.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala 40 additions, 29 deletions...scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 6 additions, 27 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/MemoryStore.scala 133 additions, 139 deletions...src/main/scala/org/apache/spark/storage/MemoryStore.scala
- core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala 172 additions, 0 deletions...la/org/apache/spark/memory/StaticMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala 17 additions, 12 deletions...g/apache/spark/storage/BlockManagerReplicationSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 22 additions, 12 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/TestShuffleMemoryManager.scala 27 additions, 1 deletion...apache/spark/sql/execution/TestShuffleMemoryManager.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala 8 additions, 5 deletions...rg/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
Loading
Please register or sign in to comment