-
- Downloads
[SPARK-10983] Unified memory manager
This patch unifies the memory management of the storage and execution regions such that either side can borrow memory from each other. When memory pressure arises, storage will be evicted in favor of execution. To avoid regressions in cases where storage is crucial, we dynamically allocate a fraction of space for storage that execution cannot evict. Several configurations are introduced: - **spark.memory.fraction (default 0.75)**: fraction of the heap space used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. - **spark.memory.storageFraction (default 0.5)**: size of the storage region within the space set aside by `spark.memory.fraction`. Cached data may only be evicted if total storage exceeds this region. - **spark.memory.useLegacyMode (default false)**: whether to use the memory management that existed in Spark 1.5 and before. This is mainly for backward compatibility. For a detailed description of the design, see [SPARK-10000](https://issues.apache.org/jira/browse/SPARK-10000). This patch builds on top of the `MemoryManager` interface introduced in #9000. Author: Andrew Or <andrew@databricks.com> Closes #9084 from andrewor14/unified-memory-manager.
Showing
- core/src/main/scala/org/apache/spark/SparkConf.scala 21 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 9 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 62 additions, 21 deletions...rc/main/scala/org/apache/spark/memory/MemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/StaticMemoryManager.scala 25 additions, 80 deletions...n/scala/org/apache/spark/memory/StaticMemoryManager.scala
- core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala 141 additions, 0 deletions.../scala/org/apache/spark/memory/UnifiedMemoryManager.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala 25 additions, 13 deletions...scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 4 additions, 0 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/MemoryStore.scala 77 additions, 44 deletions...src/main/scala/org/apache/spark/storage/MemoryStore.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 0 additions, 10 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/test/scala/org/apache/spark/DistributedSuite.scala 2 additions, 5 deletionscore/src/test/scala/org/apache/spark/DistributedSuite.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 4 additions, 2 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala 133 additions, 0 deletions...st/scala/org/apache/spark/memory/MemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala 45 additions, 60 deletions...la/org/apache/spark/memory/StaticMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala 208 additions, 0 deletions...a/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/shuffle/ShuffleMemoryManagerSuite.scala 4 additions, 1 deletion.../org/apache/spark/shuffle/ShuffleMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleSuite.scala 0 additions, 3 deletions.../org/apache/spark/shuffle/unsafe/UnsafeShuffleSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala 3 additions, 6 deletions...he/spark/util/collection/ExternalAppendOnlyMapSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala 3 additions, 20 deletions...rg/apache/spark/util/collection/ExternalSorterSuite.scala
- docs/configuration.md 70 additions, 29 deletionsdocs/configuration.md
- sql/core/src/test/scala/org/apache/spark/sql/execution/TestShuffleMemoryManager.scala 3 additions, 7 deletions...apache/spark/sql/execution/TestShuffleMemoryManager.scala
Loading
Please register or sign in to comment