-
- Downloads
[SPARK-5581][CORE] When writing sorted map output file, avoid open / …
…close between each partition ## What changes were proposed in this pull request? Replace commitAndClose with separate commit and close to avoid opening and closing the file between partitions. ## How was this patch tested? Run existing unit tests, add a few unit tests regarding reverts. Observed a ~20% reduction in total time in tasks on stages with shuffle writes to many partitions. JoshRosen Author: Brian Cho <bcho@fb.com> Closes #13382 from dafrista/separatecommit-master.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java 7 additions, 3 deletions...ache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 13 additions, 18 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java 2 additions, 1 deletion.../util/collection/unsafe/sort/UnsafeSorterSpillWriter.java
- core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala 96 additions, 61 deletions...cala/org/apache/spark/storage/DiskBlockObjectWriter.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 10 additions, 18 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 18 additions, 34 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/storage/DiskBlockObjectWriterSuite.scala 46 additions, 21 deletions...org/apache/spark/storage/DiskBlockObjectWriterSuite.scala
Loading
Please register or sign in to comment