-
- Downloads
[SPARK-9852] Let reduce tasks fetch multiple map output partitions
This makes two changes: - Allow reduce tasks to fetch multiple map output partitions -- this is a pretty small change to HashShuffleFetcher - Move shuffle locality computation out of DAGScheduler and into ShuffledRDD / MapOutputTracker; this was needed because the code in DAGScheduler wouldn't work for RDDs that fetch multiple map output partitions from each reduce task I also added an AdaptiveSchedulingSuite that creates RDDs depending on multiple map output partitions. Author: Matei Zaharia <matei@databricks.com> Closes #8844 from mateiz/spark-9852.
Showing
- core/src/main/scala/org/apache/spark/MapOutputTracker.scala 68 additions, 11 deletionscore/src/main/scala/org/apache/spark/MapOutputTracker.scala
- core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala 6 additions, 0 deletionscore/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 1 addition, 32 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/shuffle/BlockStoreShuffleReader.scala 5 additions, 4 deletions...la/org/apache/spark/shuffle/BlockStoreShuffleReader.scala
- core/src/test/scala/org/apache/spark/scheduler/AdaptiveSchedulingSuite.scala 29 additions, 18 deletions.../org/apache/spark/scheduler/AdaptiveSchedulingSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/CustomShuffledRDD.scala 111 additions, 0 deletions.../scala/org/apache/spark/scheduler/CustomShuffledRDD.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 79 additions, 58 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/shuffle/BlockStoreShuffleReaderSuite.scala 1 addition, 1 deletion...g/apache/spark/shuffle/BlockStoreShuffleReaderSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala 6 additions, 0 deletions...scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
Loading
Please register or sign in to comment