-
- Downloads
[SPARK-15958] Make initial buffer size for the Sorter configurable
## What changes were proposed in this pull request? Currently the initial buffer size in the sorter is hard coded inside the code and is too small for large workload. As a result, the sorter spends significant time expanding the buffer size and copying the data. It would be useful to have it configurable. ## How was this patch tested? Tested by running a job on the cluster. Author: Sital Kedia <skedia@fb.com> Closes #13699 from sitalkedia/config_sort_buffer_upstream.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 5 additions, 2 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java 2 additions, 2 deletions...g/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 3 additions, 1 deletion...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 5 additions, 2 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
Loading
Please register or sign in to comment