-
- Downloads
[SPARK-12081] Make unified memory manager work with small heaps
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases. **New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081). Author: Andrew Or <andrew@databricks.com> Closes #10081 from andrewor14/unified-memory-small-heaps.
Showing
- core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala 18 additions, 4 deletions.../scala/org/apache/spark/memory/UnifiedMemoryManager.scala
- core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala 20 additions, 0 deletions...a/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala
- docs/configuration.md 2 additions, 2 deletionsdocs/configuration.md
- docs/tuning.md 1 addition, 1 deletiondocs/tuning.md
Loading
Please register or sign in to comment