Skip to content
Snippets Groups Projects
Commit bd2c12fb authored by Rajesh Balamohan's avatar Rajesh Balamohan Committed by Marcelo Vanzin
Browse files

[SPARK-12920][CORE] Honor "spark.ui.retainedStages" to reduce mem-pressure 

When large number of jobs are run concurrently with Spark thrift server, thrift server starts running at high CPU due to GC pressure. Job UI retention causes memory pressure with large jobs. https://issues.apache.org/jira/secure/attachment/12783302/SPARK-12920.profiler_job_progress_listner.png has the profiler snapshot. This PR honors `spark.ui.retainedStages` strictly to reduce memory pressure.

Manual and unit tests

Author: Rajesh Balamohan <rbalamohan@apache.org>

Closes #10846 from rajeshbalamohan/SPARK-12920.
parent bf5cb8af
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment