-
- Downloads
[SPARK-13850] Force the sorter to Spill when number of elements in th…
## What changes were proposed in this pull request? Force the sorter to Spill when number of elements in the pointer array reach a certain size. This is to workaround the issue of timSort failing on large buffer size. ## How was this patch tested? Tested by running a job which was failing without this change due to TimSort bug. Author: Sital Kedia <skedia@fb.com> Closes #13107 from sitalkedia/fix_TimSort.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 7 additions, 3 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 20 additions, 3 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 3 additions, 0 deletions...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 2 additions, 0 deletions...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 3 additions, 0 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 6 additions, 2 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/WindowExec.scala 2 additions, 0 deletions...ain/scala/org/apache/spark/sql/execution/WindowExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala 4 additions, 1 deletion...che/spark/sql/execution/datasources/WriterContainer.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProductExec.scala 2 additions, 0 deletions...ache/spark/sql/execution/joins/CartesianProductExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala 4 additions, 1 deletion...apache/spark/sql/execution/streaming/FileStreamSink.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeKVExternalSorterSuite.scala 3 additions, 1 deletion...che/spark/sql/execution/UnsafeKVExternalSorterSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala 4 additions, 1 deletion...cala/org/apache/spark/sql/hive/hiveWriterContainers.scala
Loading
Please register or sign in to comment