Skip to content
Snippets Groups Projects
  • José Hiram Soltren's avatar
    640f9423
    [SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting · 640f9423
    José Hiram Soltren authored
    Builds on top of work in SPARK-8425 to update Application Level Blacklisting in the scheduler.
    
    ## What changes were proposed in this pull request?
    
    Adds a UI to these patches by:
    - defining new listener events for blacklisting and unblacklisting, nodes and executors;
    - sending said events at the relevant points in BlacklistTracker;
    - adding JSON (de)serialization code for these events;
    - augmenting the Executors UI page to show which, and how many, executors are blacklisted;
    - adding a unit test to make sure events are being fired;
    - adding HistoryServerSuite coverage to verify that the SHS reads these events correctly.
    - updates the Executor UI to show Blacklisted/Active/Dead as a tri-state in Executors Status
    
    Updates .rat-excludes to pass tests.
    
    username squito
    
    ## How was this patch tested?
    
    ./dev/run-tests
    testOnly org.apache.spark.util.JsonProtocolSuite
    testOnly org.apache.spark.scheduler.BlacklistTrackerSuite
    testOnly org.apache.spark.deploy.history.HistoryServerSuite
    https://github.com/jsoltren/jose-utils/blob/master/blacklist/test-blacklist.sh
    ![blacklist-20161219](https://cloud.githubusercontent.com/assets/1208477/21335321/9eda320a-c623-11e6-8b8c-9c912a73c276.jpg)
    
    Author: José Hiram Soltren <jose@cloudera.com>
    
    Closes #16346 from jsoltren/SPARK-16654-submit.
    640f9423
    History
    [SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting
    José Hiram Soltren authored
    Builds on top of work in SPARK-8425 to update Application Level Blacklisting in the scheduler.
    
    ## What changes were proposed in this pull request?
    
    Adds a UI to these patches by:
    - defining new listener events for blacklisting and unblacklisting, nodes and executors;
    - sending said events at the relevant points in BlacklistTracker;
    - adding JSON (de)serialization code for these events;
    - augmenting the Executors UI page to show which, and how many, executors are blacklisted;
    - adding a unit test to make sure events are being fired;
    - adding HistoryServerSuite coverage to verify that the SHS reads these events correctly.
    - updates the Executor UI to show Blacklisted/Active/Dead as a tri-state in Executors Status
    
    Updates .rat-excludes to pass tests.
    
    username squito
    
    ## How was this patch tested?
    
    ./dev/run-tests
    testOnly org.apache.spark.util.JsonProtocolSuite
    testOnly org.apache.spark.scheduler.BlacklistTrackerSuite
    testOnly org.apache.spark.deploy.history.HistoryServerSuite
    https://github.com/jsoltren/jose-utils/blob/master/blacklist/test-blacklist.sh
    ![blacklist-20161219](https://cloud.githubusercontent.com/assets/1208477/21335321/9eda320a-c623-11e6-8b8c-9c912a73c276.jpg)
    
    Author: José Hiram Soltren <jose@cloudera.com>
    
    Closes #16346 from jsoltren/SPARK-16654-submit.