-
- Downloads
[SPARK-15083][WEB UI] History Server can OOM due to unlimited TaskUIData
## What changes were proposed in this pull request? Based on #12990 by tankkyo Since the History Server currently loads all application's data it can OOM if too many applications have a significant task count. `spark.ui.trimTasks` (default: false) can be set to true to trim tasks by `spark.ui.retainedTasks` (default: 10000) (This is a "quick fix" to help those running into the problem until a update of how the history server loads app data can be done) ## How was this patch tested? Manual testing and dev/run-tests  Author: Alex Bozarth <ajbozart@us.ibm.com> Closes #14673 from ajbozarth/spark15083.
Showing
- core/src/main/scala/org/apache/spark/internal/config/package.scala 5 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala 8 additions, 1 deletion.../scala/org/apache/spark/ui/jobs/JobProgressListener.scala
- core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala 10 additions, 2 deletionscore/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_expectation.json 65 additions, 65 deletions...erExpectations/stage_task_list_w__sortBy_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_short_names___runtime_expectation.json 65 additions, 65 deletions...ask_list_w__sortBy_short_names___runtime_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_short_names__runtime_expectation.json 91 additions, 91 deletions...task_list_w__sortBy_short_names__runtime_expectation.json
- core/src/test/scala/org/apache/spark/status/api/v1/AllStagesResourceSuite.scala 2 additions, 2 deletions...g/apache/spark/status/api/v1/AllStagesResourceSuite.scala
- docs/configuration.md 8 additions, 0 deletionsdocs/configuration.md
Loading
Please register or sign in to comment