Skip to content
  • Yangyang Liu's avatar
    68df47ac
    [SPARK-16405] Add metrics and source for external shuffle service · 68df47ac
    Yangyang Liu authored
    ## What changes were proposed in this pull request?
    
    Since externalShuffleService is essential for spark, better monitoring for shuffle service is necessary. In order to do so, we added various metrics in shuffle service and imported into ExternalShuffleServiceSource for metric system.
    Metrics added in shuffle service:
    * registeredExecutorsSize
    * openBlockRequestLatencyMillis
    * registerExecutorRequestLatencyMillis
    * blockTransferRateBytes
    
    JIRA Issue: https://issues.apache.org/jira/browse/SPARK-16405
    
    ## How was this patch tested?
    
    Some test cases are added to verify metrics as expected in metric system. Those unit test cases are shown in `ExternalShuffleBlockHandlerSuite `
    
    Author: Yangyang Liu <yangyangliu@fb.com>
    
    Closes #14080 from lovexi/yangyang-metrics.
    68df47ac
    [SPARK-16405] Add metrics and source for external shuffle service
    Yangyang Liu authored
    ## What changes were proposed in this pull request?
    
    Since externalShuffleService is essential for spark, better monitoring for shuffle service is necessary. In order to do so, we added various metrics in shuffle service and imported into ExternalShuffleServiceSource for metric system.
    Metrics added in shuffle service:
    * registeredExecutorsSize
    * openBlockRequestLatencyMillis
    * registerExecutorRequestLatencyMillis
    * blockTransferRateBytes
    
    JIRA Issue: https://issues.apache.org/jira/browse/SPARK-16405
    
    ## How was this patch tested?
    
    Some test cases are added to verify metrics as expected in metric system. Those unit test cases are shown in `ExternalShuffleBlockHandlerSuite `
    
    Author: Yangyang Liu <yangyangliu@fb.com>
    
    Closes #14080 from lovexi/yangyang-metrics.
Loading