-
- Downloads
[SPARK-20950][CORE] add a new config to diskWriteBufferSize which is hard coded before
## What changes were proposed in this pull request? This PR Improvement in two: 1.With spark.shuffle.spill.diskWriteBufferSize configure diskWriteBufferSize of ShuffleExternalSorter. when change the size of the diskWriteBufferSize to test `forceSorterToSpill` The average performance of running 10 times is as follows:(their unit is MS). ``` diskWriteBufferSize: 1M 512K 256K 128K 64K 32K 16K 8K 4K --------------------------------------------------------------------------------------- RecordSize = 2.5M 742 722 694 686 667 668 671 669 683 RecordSize = 1M 294 293 292 287 283 285 281 279 285 ``` 2.Remove outputBufferSizeInBytes and inputBufferSizeInBytes to initialize in mergeSpillsWithFileStream function. ## How was this patch tested? The unit test. Author: caoxuewen <cao.xuewen@zte.com.cn> Closes #18174 from heary-cao/buffersize.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 8 additions, 3 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 10 additions, 4 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java 15 additions, 9 deletions.../util/collection/unsafe/sort/UnsafeSorterSpillWriter.java
- core/src/main/scala/org/apache/spark/internal/config/package.scala 27 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
Loading
Please register or sign in to comment