-
- Downloads
[SPARK-15317][CORE] Don't store accumulators for every task in listeners
## What changes were proposed in this pull request? In general, the Web UI doesn't need to store the Accumulator/AccumulableInfo for every task. It only needs the Accumulator values. In this PR, it creates new UIData classes to store the necessary fields and make `JobProgressListener` store only these new classes, so that `JobProgressListener` won't store Accumulator/AccumulableInfo and the size of `JobProgressListener` becomes pretty small. I also eliminates `AccumulableInfo` from `SQLListener` so that we don't keep any references for those unused `AccumulableInfo`s. ## How was this patch tested? I ran two tests reported in JIRA locally: The first one is: ``` val data = spark.range(0, 10000, 1, 10000) data.cache().count() ``` The retained size of JobProgressListener decreases from 60.7M to 6.9M. The second one is: ``` import org.apache.spark.ml.CC import org.apache.spark.sql.SQLContext val sqlContext = SQLContext.getOrCreate(sc) CC.runTest(sqlContext) ``` This test won't cause OOM after applying this patch. Author: Shixiong Zhu <shixiong@databricks.com> Closes #13153 from zsxwing/memory.
Showing
- core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala 1 addition, 1 deletion...la/org/apache/spark/status/api/v1/AllStagesResource.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 6 additions, 6 deletions.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 131 additions, 5 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala 3 additions, 0 deletions.../src/main/scala/org/apache/spark/util/AccumulatorV2.scala
- core/src/test/scala/org/apache/spark/status/api/v1/AllStagesResourceSuite.scala 2 additions, 2 deletions...g/apache/spark/status/api/v1/AllStagesResourceSuite.scala
- core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala 28 additions, 1 deletion...a/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 3 additions, 5 deletions...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala 7 additions, 6 deletions...scala/org/apache/spark/sql/execution/ui/SQLListener.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 3 additions, 4 deletions...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala 1 addition, 1 deletion.../org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
Loading
Please register or sign in to comment