-
- Downloads
[SPARK-14753][CORE] remove internal flag in Accumulable
## What changes were proposed in this pull request? the `Accumulable.internal` flag is only used to avoid registering internal accumulators for 2 certain cases: 1. `TaskMetrics.createTempShuffleReadMetrics`: the accumulators in the temp shuffle read metrics should not be registered. 2. `TaskMetrics.fromAccumulatorUpdates`: the created task metrics is only used to post event, accumulators inside it should not be registered. For 1, we can create a `TempShuffleReadMetrics` that don't create accumulators, just keep the data and merge it at last. For 2, we can un-register these accumulators immediately. TODO: remove `internal` flag in `AccumulableInfo` with followup PR ## How was this patch tested? existing tests. Author: Wenchen Fan <wenchen@databricks.com> Closes #12525 from cloud-fan/acc.
Showing
- core/src/main/scala/org/apache/spark/Accumulable.scala 8 additions, 35 deletionscore/src/main/scala/org/apache/spark/Accumulable.scala
- core/src/main/scala/org/apache/spark/Accumulator.scala 3 additions, 14 deletionscore/src/main/scala/org/apache/spark/Accumulator.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 2 additions, 1 deletioncore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala 29 additions, 2 deletions.../scala/org/apache/spark/executor/ShuffleReadMetrics.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 20 additions, 11 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/Stage.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/scheduler/Stage.scala
- core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala 2 additions, 2 deletions...src/main/scala/org/apache/spark/scheduler/StageInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 2 additions, 3 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 1 addition, 1 deletion.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/AccumulatorSuite.scala 8 additions, 24 deletionscore/src/test/scala/org/apache/spark/AccumulatorSuite.scala
- core/src/test/scala/org/apache/spark/executor/TaskMetricsSuite.scala 14 additions, 19 deletions...st/scala/org/apache/spark/executor/TaskMetricsSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala 4 additions, 4 deletions...t/scala/org/apache/spark/scheduler/TaskContextSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 2 additions, 3 deletions...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
Loading
Please register or sign in to comment