Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    23a9e62b
    [SPARK-12251] Document and improve off-heap memory configurations · 23a9e62b
    Josh Rosen authored
    This patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those configs.
    
    - Change `spark.memory.offHeapSize` to `spark.memory.offHeap.size`. This is fine because this configuration has not shipped in any Spark release yet (it's new in Spark 1.6).
    - Deprecated `spark.unsafe.offHeap` in favor of a new `spark.memory.offHeap.enabled` configuration. The motivation behind this change is to gather all memory-related configurations under the same prefix.
    - Add a check which prevents users from setting `spark.memory.offHeap.enabled=true` when `spark.memory.offHeap.size == 0`. After SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard limit on the amount of off-heap memory that it will allocate to tasks. As a result, enabling off-heap execution memory without setting `spark.memory.offHeap.size` will lead to immediate OOMs. The new configuration validation makes this scenario easier to diagnose, helping to avoid user confusion.
    - Document these configurations on the configuration page.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #10237 from JoshRosen/SPARK-12251.
    23a9e62b
    History
    [SPARK-12251] Document and improve off-heap memory configurations
    Josh Rosen authored
    This patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those configs.
    
    - Change `spark.memory.offHeapSize` to `spark.memory.offHeap.size`. This is fine because this configuration has not shipped in any Spark release yet (it's new in Spark 1.6).
    - Deprecated `spark.unsafe.offHeap` in favor of a new `spark.memory.offHeap.enabled` configuration. The motivation behind this change is to gather all memory-related configurations under the same prefix.
    - Add a check which prevents users from setting `spark.memory.offHeap.enabled=true` when `spark.memory.offHeap.size == 0`. After SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard limit on the amount of off-heap memory that it will allocate to tasks. As a result, enabling off-heap execution memory without setting `spark.memory.offHeap.size` will lead to immediate OOMs. The new configuration validation makes this scenario easier to diagnose, helping to avoid user confusion.
    - Document these configurations on the configuration page.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #10237 from JoshRosen/SPARK-12251.