Skip to content
Snippets Groups Projects
  • jerryshao's avatar
    a4491626
    [SPARK-17019][CORE] Expose on-heap and off-heap memory usage in various places · a4491626
    jerryshao authored
    ## What changes were proposed in this pull request?
    
    With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992), Spark supports persisting data into off-heap memory, but the usage of on-heap and off-heap memory is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:
    1. Spark UI's executor page will display both on-heap and off-heap memory usage.
    2. REST request returns both on-heap and off-heap memory.
    3. Also this can be gotten from MetricsSystem.
    4. Last this usage can be obtained programmatically from SparkListener.
    
    Attach the UI changes:
    
    ![screen shot 2016-08-12 at 11 20 44 am](https://cloud.githubusercontent.com/assets/850797/17612032/6c2f4480-607f-11e6-82e8-a27fb8cbb4ae.png)
    
    Backward compatibility is also considered for event-log and REST API. Old event log can still be replayed with off-heap usage displayed as 0. For REST API, only adds the new fields, so JSON backward compatibility can still be kept.
    ## How was this patch tested?
    
    Unit test added and manual verification.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #14617 from jerryshao/SPARK-17019.
    a4491626
    History
    [SPARK-17019][CORE] Expose on-heap and off-heap memory usage in various places
    jerryshao authored
    ## What changes were proposed in this pull request?
    
    With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992), Spark supports persisting data into off-heap memory, but the usage of on-heap and off-heap memory is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:
    1. Spark UI's executor page will display both on-heap and off-heap memory usage.
    2. REST request returns both on-heap and off-heap memory.
    3. Also this can be gotten from MetricsSystem.
    4. Last this usage can be obtained programmatically from SparkListener.
    
    Attach the UI changes:
    
    ![screen shot 2016-08-12 at 11 20 44 am](https://cloud.githubusercontent.com/assets/850797/17612032/6c2f4480-607f-11e6-82e8-a27fb8cbb4ae.png)
    
    Backward compatibility is also considered for event-log and REST API. Old event log can still be replayed with off-heap usage displayed as 0. For REST API, only adds the new fields, so JSON backward compatibility can still be kept.
    ## How was this patch tested?
    
    Unit test added and manual verification.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #14617 from jerryshao/SPARK-17019.