-
- Downloads
[SPARK-12295] [SQL] external spilling for window functions
This PR manage the memory used by window functions (buffered rows), also enable external spilling. After this PR, we can run window functions on a partition with hundreds of millions of rows with only 1G. Author: Davies Liu <davies@databricks.com> Closes #10605 from davies/unsafe_window.
Showing
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 20 additions, 1 deletion...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 12 additions, 6 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterIterator.java 2 additions, 0 deletions...ark/util/collection/unsafe/sort/UnsafeSorterIterator.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillMerger.java 7 additions, 0 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillMerger.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java 7 additions, 1 deletion.../util/collection/unsafe/sort/UnsafeSorterSpillReader.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala 228 additions, 86 deletions...rc/main/scala/org/apache/spark/sql/execution/Window.scala
Loading
Please register or sign in to comment