-
- Downloads
[SPARK-12251] Document and improve off-heap memory configurations
This patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those configs. - Change `spark.memory.offHeapSize` to `spark.memory.offHeap.size`. This is fine because this configuration has not shipped in any Spark release yet (it's new in Spark 1.6). - Deprecated `spark.unsafe.offHeap` in favor of a new `spark.memory.offHeap.enabled` configuration. The motivation behind this change is to gather all memory-related configurations under the same prefix. - Add a check which prevents users from setting `spark.memory.offHeap.enabled=true` when `spark.memory.offHeap.size == 0`. After SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard limit on the amount of off-heap memory that it will allocate to tasks. As a result, enabling off-heap execution memory without setting `spark.memory.offHeap.size` will lead to immediate OOMs. The new configuration validation makes this scenario easier to diagnose, helping to avoid user confusion. - Document these configurations on the configuration page. Author: Josh Rosen <joshrosen@databricks.com> Closes #10237 from JoshRosen/SPARK-12251.
Showing
- core/src/main/scala/org/apache/spark/SparkConf.scala 3 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/memory/MemoryManager.scala 8 additions, 2 deletions...rc/main/scala/org/apache/spark/memory/MemoryManager.scala
- core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java 17 additions, 4 deletions.../java/org/apache/spark/memory/TaskMemoryManagerSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/PackedRecordPointerSuite.java 4 additions, 2 deletions...g/apache/spark/shuffle/sort/PackedRecordPointerSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java 2 additions, 2 deletions...apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java 1 addition, 1 deletion...g/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 2 additions, 2 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 1 addition, 1 deletion...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorterSuite.java 2 additions, 2 deletions...til/collection/unsafe/sort/UnsafeInMemorySorterSuite.java
- core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala 1 addition, 1 deletion...la/org/apache/spark/memory/StaticMemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala 1 addition, 1 deletion...a/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala
- docs/configuration.md 16 additions, 0 deletionsdocs/configuration.md
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 5 additions, 1 deletion...org/apache/spark/sql/execution/joins/HashedRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 1 addition, 1 deletion...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeKVExternalSorterSuite.scala 1 addition, 1 deletion...che/spark/sql/execution/UnsafeKVExternalSorterSuite.scala
Loading
Please register or sign in to comment