-
- Downloads
[SPARK-3613] Record only average block size in MapStatus for large stages
This changes the way we send MapStatus from executors back to driver for large stages (>2000 tasks). For large stages, we no longer send one byte per block. Instead, we just send the average block size. This makes large jobs (tens of thousands of tasks) much more reliable since the driver no longer sends huge amount of data. Author: Reynold Xin <rxin@apache.org> Closes #2470 from rxin/mapstatus and squashes the following commits: 822ff54 [Reynold Xin] Code review feedback. 3b86f56 [Reynold Xin] Added MimaExclude. f89d182 [Reynold Xin] Fixed a bug in MapStatus 6a0401c [Reynold Xin] [SPARK-3613] Record only average block size in MapStatus for large stages.
Showing
- core/src/main/scala/org/apache/spark/MapOutputTracker.scala 1 addition, 28 deletionscore/src/main/scala/org/apache/spark/MapOutputTracker.scala
- core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala 110 additions, 9 deletions...src/main/scala/org/apache/spark/scheduler/MapStatus.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala 3 additions, 5 deletions...ala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala 1 addition, 2 deletions...ala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
- core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala 22 additions, 44 deletions...c/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/MapStatusSuite.scala 92 additions, 0 deletions...est/scala/org/apache/spark/scheduler/MapStatusSuite.scala
- core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala 6 additions, 8 deletions...src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala
- project/MimaExcludes.scala 4 additions, 1 deletionproject/MimaExcludes.scala
Loading
Please register or sign in to comment