-
- Downloads
[SPARK-9531] [SQL] UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter
This pull request adds a destructAndCreateExternalSorter method to UnsafeFixedWidthAggregationMap. The new method does the following: 1. Creates a new external sorter UnsafeKVExternalSorter 2. Adds all the data into an in-memory sorter, sorts them 3. Spills the sorted in-memory data to disk This method can be used to fallback to sort-based aggregation when under memory pressure. The pull request also includes accounting fixes from JoshRosen. TODOs (that can be done in follow-up PRs) - [x] Address Josh's feedbacks from #7849 - [x] More documentation and test cases - [x] Make sure we are doing memory accounting correctly with test cases (e.g. did we release the memory in BytesToBytesMap twice?) - [ ] Look harder at possible memory leaks and exception handling - [ ] Randomized tester for the KV sorter as well as the aggregation map Author: Reynold Xin <rxin@databricks.com> Author: Josh Rosen <joshrosen@databricks.com> Closes #7860 from rxin/kvsorter and squashes the following commits: 986a58c [Reynold Xin] Bug fix. 599317c [Reynold Xin] Style fix and slightly more compact code. fe7bd4e [Reynold Xin] Bug fixes. fd71bef [Reynold Xin] Merge remote-tracking branch 'josh/large-records-in-sql-sorter' into kvsorter-with-josh-fix 3efae38 [Reynold Xin] More fixes and documentation. 45f1b09 [Josh Rosen] Ensure that spill files are cleaned up f6a9bd3 [Reynold Xin] Josh feedback. 9be8139 [Reynold Xin] Remove testSpillFrequency. 7cbe759 [Reynold Xin] [SPARK-9531][SQL] UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter. ae4a8af [Josh Rosen] Detect leaked unsafe memory in UnsafeExternalSorterSuite. 52f9b06 [Josh Rosen] Detect ShuffleMemoryManager leaks in UnsafeExternalSorter.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 24 additions, 8 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 152 additions, 45 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java 4 additions, 0 deletions...ark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java 3 additions, 0 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillReader.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java 4 additions, 0 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillWriter.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 3 additions, 4 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 47 additions, 18 deletions...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java 2 additions, 1 deletion.../org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 5 additions, 4 deletions...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/core/pom.xml 5 additions, 0 deletionssql/core/pom.xml
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 11 additions, 92 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 236 additions, 0 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/SortPrefixUtils.scala 26 additions, 7 deletions...cala/org/apache/spark/sql/execution/SortPrefixUtils.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/sort.scala 3 additions, 1 deletion.../src/main/scala/org/apache/spark/sql/execution/sort.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/TestShuffleMemoryManager.scala 51 additions, 0 deletions...apache/spark/sql/execution/TestShuffleMemoryManager.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 89 additions, 35 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeKVExternalSorterSuite.scala 158 additions, 0 deletions...che/spark/sql/execution/UnsafeKVExternalSorterSuite.scala
Loading
Please register or sign in to comment