-
- Downloads
[SPARK-12837][CORE] reduce network IO for accumulators
Sending un-updated accumulators back to driver makes no sense, as merging a zero value accumulator is a no-op. We should only send back updated accumulators, to save network IO. new test in `TaskContextSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #12899 from cloud-fan/acc.
Showing
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 8 additions, 1 deletioncore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala 2 additions, 2 deletions.../src/main/scala/org/apache/spark/util/AccumulatorV2.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala 30 additions, 7 deletions...t/scala/org/apache/spark/scheduler/TaskContextSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 3 additions, 3 deletions...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/ui/SQLListener.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala 6 additions, 6 deletions.../org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
Loading
Please register or sign in to comment