-
- Downloads
[SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting
Builds on top of work in SPARK-8425 to update Application Level Blacklisting in the scheduler. ## What changes were proposed in this pull request? Adds a UI to these patches by: - defining new listener events for blacklisting and unblacklisting, nodes and executors; - sending said events at the relevant points in BlacklistTracker; - adding JSON (de)serialization code for these events; - augmenting the Executors UI page to show which, and how many, executors are blacklisted; - adding a unit test to make sure events are being fired; - adding HistoryServerSuite coverage to verify that the SHS reads these events correctly. - updates the Executor UI to show Blacklisted/Active/Dead as a tri-state in Executors Status Updates .rat-excludes to pass tests. username squito ## How was this patch tested? ./dev/run-tests testOnly org.apache.spark.util.JsonProtocolSuite testOnly org.apache.spark.scheduler.BlacklistTrackerSuite testOnly org.apache.spark.deploy.history.HistoryServerSuite https://github.com/jsoltren/jose-utils/blob/master/blacklist/test-blacklist.sh  Author: José Hiram Soltren <jose@cloudera.com> Closes #16346 from jsoltren/SPARK-16654-submit.
Showing
- core/src/main/java/org/apache/spark/SparkFirehoseListener.java 20 additions, 0 deletions...src/main/java/org/apache/spark/SparkFirehoseListener.java
- core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html 5 additions, 0 deletions...es/org/apache/spark/ui/static/executorspage-template.html
- core/src/main/resources/org/apache/spark/ui/static/executorspage.js 27 additions, 12 deletions...ain/resources/org/apache/spark/ui/static/executorspage.js
- core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala 14 additions, 2 deletions...n/scala/org/apache/spark/scheduler/BlacklistTracker.scala
- core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala 16 additions, 0 deletions...ala/org/apache/spark/scheduler/EventLoggingListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 54 additions, 0 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 8 additions, 0 deletions...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 5 additions, 5 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/status/api/v1/api.scala 1 addition, 0 deletionscore/src/main/scala/org/apache/spark/status/api/v1/api.scala
- core/src/main/scala/org/apache/spark/ui/ToolTips.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/ui/ToolTips.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala 2 additions, 0 deletions...c/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala 52 additions, 5 deletions...rc/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
- core/src/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala 6 additions, 0 deletions...c/main/scala/org/apache/spark/ui/jobs/ExecutorTable.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 2 additions, 0 deletionscore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json 36 additions, 8 deletions...ServerExpectations/application_list_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json 36 additions, 8 deletions...rverExpectations/completed_app_list_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json 1 addition, 0 deletions...oryServerExpectations/executor_list_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/executor_node_blacklisting_expectation.json 118 additions, 0 deletions...rExpectations/executor_node_blacklisting_expectation.json
- core/src/test/resources/HistoryServerExpectations/executor_node_blacklisting_unblacklisting_expectation.json 118 additions, 0 deletions...xecutor_node_blacklisting_unblacklisting_expectation.json
- core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json 20 additions, 44 deletions...ryServerExpectations/limit_app_list_json_expectation.json
Loading
Please register or sign in to comment