-
- Downloads
[SPARK-12895][SPARK-12896] Migrate TaskMetrics to accumulators
The high level idea is that instead of having the executors send both accumulator updates and TaskMetrics, we should have them send only accumulator updates. This eliminates the need to maintain both code paths since one can be implemented in terms of the other. This effort is split into two parts: **SPARK-12895: Implement TaskMetrics using accumulators.** TaskMetrics is basically just a bunch of accumulable fields. This patch makes TaskMetrics a syntactic wrapper around a collection of accumulators so we don't need to send TaskMetrics from the executors to the driver. **SPARK-12896: Send only accumulator updates to the driver.** Now that TaskMetrics are expressed in terms of accumulators, we can capture all TaskMetrics values if we just send accumulator updates from the executors to the driver. This completes the parent issue SPARK-10620. While an effort has been made to preserve as much of the public API as possible, there were a few known breaking DeveloperApi changes that would be very awkward to maintain. I will gather the full list shortly and post it here. Note: This was once part of #10717. This patch is split out into its own patch from there to make it easier for others to review. Other smaller pieces of already been merged into master. Author: Andrew Or <andrew@databricks.com> Closes #10835 from andrewor14/task-metrics-use-accums.
Showing
- core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java 1 addition, 7 deletions...va/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java
- core/src/main/scala/org/apache/spark/Accumulable.scala 63 additions, 23 deletionscore/src/main/scala/org/apache/spark/Accumulable.scala
- core/src/main/scala/org/apache/spark/Accumulator.scala 74 additions, 27 deletionscore/src/main/scala/org/apache/spark/Accumulator.scala
- core/src/main/scala/org/apache/spark/Aggregator.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/Aggregator.scala
- core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/HeartbeatReceiver.scala
- core/src/main/scala/org/apache/spark/InternalAccumulator.scala 175 additions, 24 deletions...src/main/scala/org/apache/spark/InternalAccumulator.scala
- core/src/main/scala/org/apache/spark/TaskContext.scala 1 addition, 18 deletionscore/src/main/scala/org/apache/spark/TaskContext.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 9 additions, 21 deletionscore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/TaskEndReason.scala 22 additions, 7 deletionscore/src/main/scala/org/apache/spark/TaskEndReason.scala
- core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala 8 additions, 0 deletions.../main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 19 additions, 26 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/InputMetrics.scala 57 additions, 24 deletions...c/main/scala/org/apache/spark/executor/InputMetrics.scala
- core/src/main/scala/org/apache/spark/executor/OutputMetrics.scala 61 additions, 10 deletions.../main/scala/org/apache/spark/executor/OutputMetrics.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala 68 additions, 36 deletions.../scala/org/apache/spark/executor/ShuffleReadMetrics.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleWriteMetrics.scala 44 additions, 18 deletions...scala/org/apache/spark/executor/ShuffleWriteMetrics.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 235 additions, 135 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 16 additions, 8 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 14 additions, 8 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/AccumulableInfo.scala 33 additions, 22 deletions...in/scala/org/apache/spark/scheduler/AccumulableInfo.scala
Loading
Please register or sign in to comment