Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    f6d06adf
    [SPARK-10708] Consolidate sort shuffle implementations · f6d06adf
    Josh Rosen authored
    There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #8829 from JoshRosen/consolidate-sort-shuffle-implementations.
    f6d06adf
    History
    [SPARK-10708] Consolidate sort shuffle implementations
    Josh Rosen authored
    There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #8829 from JoshRosen/consolidate-sort-shuffle-implementations.