-
- Downloads
[SPARK-21595] Separate thresholds for buffering and spilling in ExternalAppendOnlyUnsafeRowArray
## What changes were proposed in this pull request? [SPARK-21595](https://issues.apache.org/jira/browse/SPARK-21595) reported that there is excessive spilling to disk due to default spill threshold for `ExternalAppendOnlyUnsafeRowArray` being quite small for WINDOW operator. Old behaviour of WINDOW operator (pre https://github.com/apache/spark/pull/16909) would hold data in an array for first 4096 records post which it would switch to `UnsafeExternalSorter` and start spilling to disk after reaching `spark.shuffle.spill.numElementsForceSpillThreshold` (or earlier if there was paucity of memory due to excessive consumers). Currently the (switch from in-memory to `UnsafeExternalSorter`) and (`UnsafeExternalSorter` spilling to disk) for `ExternalAppendOnlyUnsafeRowArray` is controlled by a single threshold. This PR aims to separate that to have more granular control. ## How was this patch tested? Added unit tests Author: Tejas Patil <tejasp@fb.com> Closes #18843 from tejasapatil/SPARK-21595.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 36 additions, 5 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArray.scala 15 additions, 13 deletions...park/sql/execution/ExternalAppendOnlyUnsafeRowArray.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProductExec.scala 8 additions, 4 deletions...ache/spark/sql/execution/joins/CartesianProductExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala 21 additions, 3 deletions.../apache/spark/sql/execution/joins/SortMergeJoinExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala 3 additions, 1 deletion...la/org/apache/spark/sql/execution/window/WindowExec.scala
- sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala 2 additions, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala 5 additions, 2 deletions...execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArraySuite.scala 63 additions, 40 deletions...sql/execution/ExternalAppendOnlyUnsafeRowArraySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala 2 additions, 1 deletion...g/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
Loading
Please register or sign in to comment