-
- Downloads
[SPARK-2124] Move aggregation into shuffle implementations
This PR is a sub-task of SPARK-2044 to move the execution of aggregation into shuffle implementations. I leave `CoGoupedRDD` and `SubtractedRDD` unchanged because they have their implementations of aggregation. I'm not sure is it suitable to change these two RDDs. Also I do not move sort related code of `OrderedRDDFunctions` into shuffle, this will be solved in another sub-task. Author: jerryshao <saisai.shao@intel.com> Closes #1064 from jerryshao/SPARK-2124 and squashes the following commits: 4a05a40 [jerryshao] Modify according to comments 1f7dcc8 [jerryshao] Style changes 50a2fd6 [jerryshao] Fix test suite issue after moving aggregator to Shuffle reader and writer 1a96190 [jerryshao] Code modification related to the ShuffledRDD 308f635 [jerryshao] initial works of move combiner to ShuffleManager's reader and writer
Showing
- core/src/main/scala/org/apache/spark/Dependency.scala 2 additions, 1 deletioncore/src/main/scala/org/apache/spark/Dependency.scala
- core/src/main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala 1 addition, 1 deletion...main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala
- core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala 5 additions, 15 deletions...rc/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala 31 additions, 6 deletionscore/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala 2 additions, 4 deletions...ain/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleWriter.scala 2 additions, 2 deletions...c/main/scala/org/apache/spark/shuffle/ShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala 17 additions, 3 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleReader.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala 18 additions, 5 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala
- core/src/test/scala/org/apache/spark/CheckpointSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/CheckpointSuite.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 14 additions, 8 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala 7 additions, 7 deletionscore/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala 1 addition, 1 deletion...scala/org/apache/spark/scheduler/SparkListenerSuite.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/MessageToPartition.scala 4 additions, 3 deletions...ala/org/apache/spark/graphx/impl/MessageToPartition.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala 2 additions, 2 deletions.../org/apache/spark/graphx/impl/RoutingTablePartition.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala 3 additions, 3 deletions.../main/scala/org/apache/spark/sql/execution/Exchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/basicOperators.scala
Loading
Please register or sign in to comment