-
- Downloads
[SPARK-14704][CORE] create accumulators in TaskMetrics
## What changes were proposed in this pull request? Before this PR, we create accumulators at driver side(and register them) and send them to executor side, then we create `TaskMetrics` with these accumulators at executor side. After this PR, we will create `TaskMetrics` at driver side and send it to executor side, so that we can create accumulators inside `TaskMetrics` directly, which is cleaner. ## How was this patch tested? existing tests. Author: Wenchen Fan <wenchen@databricks.com> Closes #12472 from cloud-fan/acc.
Showing
- core/src/main/scala/org/apache/spark/InternalAccumulator.scala 0 additions, 127 deletions...src/main/scala/org/apache/spark/InternalAccumulator.scala
- core/src/main/scala/org/apache/spark/TaskContext.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/TaskContext.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 1 addition, 6 deletionscore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 13 additions, 19 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/InputMetrics.scala 5 additions, 9 deletions...c/main/scala/org/apache/spark/executor/InputMetrics.scala
- core/src/main/scala/org/apache/spark/executor/OutputMetrics.scala 6 additions, 9 deletions.../main/scala/org/apache/spark/executor/OutputMetrics.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala 16 additions, 33 deletions.../scala/org/apache/spark/executor/ShuffleReadMetrics.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleWriteMetrics.scala 9 additions, 25 deletions...scala/org/apache/spark/executor/ShuffleWriteMetrics.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 75 additions, 100 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 2 additions, 2 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/ResultTask.scala 4 additions, 6 deletions...rc/main/scala/org/apache/spark/scheduler/ResultTask.scala
- core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala 4 additions, 6 deletions...ain/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala 4 additions, 4 deletions...src/main/scala/org/apache/spark/scheduler/StageInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 7 additions, 9 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 17 additions, 24 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/AccumulatorSuite.scala 7 additions, 11 deletionscore/src/test/scala/org/apache/spark/AccumulatorSuite.scala
- core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala 1 addition, 1 deletion...ala/org/apache/spark/ExecutorAllocationManagerSuite.scala
- core/src/test/scala/org/apache/spark/InternalAccumulatorSuite.scala 10 additions, 119 deletions...est/scala/org/apache/spark/InternalAccumulatorSuite.scala
Loading
Please register or sign in to comment