-
- Downloads
[SPARK-10708] Consolidate sort shuffle implementations
There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together. Author: Josh Rosen <joshrosen@databricks.com> Closes #8829 from JoshRosen/consolidate-sort-shuffle-implementations.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java 81 additions, 25 deletions...ache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java
- core/src/main/java/org/apache/spark/shuffle/sort/PackedRecordPointer.java 1 addition, 1 deletion...va/org/apache/spark/shuffle/sort/PackedRecordPointer.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java 20 additions, 8 deletions.../org/apache/spark/shuffle/sort/ShuffleExternalSorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java 8 additions, 8 deletions.../org/apache/spark/shuffle/sort/ShuffleInMemorySorter.java
- core/src/main/java/org/apache/spark/shuffle/sort/ShuffleSortDataFormat.java 4 additions, 4 deletions.../org/apache/spark/shuffle/sort/ShuffleSortDataFormat.java
- core/src/main/java/org/apache/spark/shuffle/sort/SortShuffleFileWriter.java 0 additions, 53 deletions.../org/apache/spark/shuffle/sort/SortShuffleFileWriter.java
- core/src/main/java/org/apache/spark/shuffle/sort/SpillInfo.java 2 additions, 2 deletions...rc/main/java/org/apache/spark/shuffle/sort/SpillInfo.java
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 6 additions, 6 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/main/scala/org/apache/spark/SparkEnv.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 157 additions, 18 deletions...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 10 additions, 18 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleManager.scala 0 additions, 202 deletions...rg/apache/spark/shuffle/unsafe/UnsafeShuffleManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala 0 additions, 146 deletions...cala/org/apache/spark/util/collection/ChainedBuffer.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 7 additions, 28 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala 0 additions, 273 deletions...ark/util/collection/PartitionedSerializedPairBuffer.scala
- core/src/test/java/org/apache/spark/shuffle/sort/PackedRecordPointerSuite.java 3 additions, 2 deletions...g/apache/spark/shuffle/sort/PackedRecordPointerSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java 8 additions, 8 deletions...apache/spark/shuffle/sort/ShuffleInMemorySorterSuite.java
- core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java 5 additions, 5 deletions...g/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
- core/src/test/scala/org/apache/spark/SortShuffleSuite.scala 65 additions, 0 deletionscore/src/test/scala/org/apache/spark/SortShuffleSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 3 additions, 3 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
Loading
Please register or sign in to comment