-
- Downloads
[SPARK-17019][CORE] Expose on-heap and off-heap memory usage in various places
## What changes were proposed in this pull request? With [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992), Spark supports persisting data into off-heap memory, but the usage of on-heap and off-heap memory is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places: 1. Spark UI's executor page will display both on-heap and off-heap memory usage. 2. REST request returns both on-heap and off-heap memory. 3. Also this can be gotten from MetricsSystem. 4. Last this usage can be obtained programmatically from SparkListener. Attach the UI changes:  Backward compatibility is also considered for event-log and REST API. Old event log can still be replayed with off-heap usage displayed as 0. For REST API, only adds the new fields, so JSON backward compatibility can still be kept. ## How was this patch tested? Unit test added and manual verification. Author: jerryshao <sshao@hortonworks.com> Closes #14617 from jerryshao/SPARK-17019.
Showing
- core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html 17 additions, 1 deletion...es/org/apache/spark/ui/static/executorspage-template.html
- core/src/main/resources/org/apache/spark/ui/static/executorspage.js 101 additions, 2 deletions...ain/resources/org/apache/spark/ui/static/executorspage.js
- core/src/main/resources/org/apache/spark/ui/static/webui.css 2 additions, 1 deletioncore/src/main/resources/org/apache/spark/ui/static/webui.css
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 7 additions, 2 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/status/api/v1/AllRDDResource.scala 7 additions, 1 deletion...scala/org/apache/spark/status/api/v1/AllRDDResource.scala
- core/src/main/scala/org/apache/spark/status/api/v1/api.scala 10 additions, 2 deletionscore/src/main/scala/org/apache/spark/status/api/v1/api.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 5 additions, 4 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 3 additions, 2 deletions...n/scala/org/apache/spark/storage/BlockManagerMaster.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala 14 additions, 8 deletions...org/apache/spark/storage/BlockManagerMasterEndpoint.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala 2 additions, 1 deletion...scala/org/apache/spark/storage/BlockManagerMessages.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerSource.scala 35 additions, 31 deletions...n/scala/org/apache/spark/storage/BlockManagerSource.scala
- core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala 5 additions, 3 deletions...cala/org/apache/spark/storage/StorageStatusListener.scala
- core/src/main/scala/org/apache/spark/storage/StorageUtils.scala 68 additions, 31 deletions...rc/main/scala/org/apache/spark/storage/StorageUtils.scala
- core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala 44 additions, 2 deletions...c/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
- core/src/main/scala/org/apache/spark/ui/storage/RDDPage.scala 8 additions, 3 deletions.../src/main/scala/org/apache/spark/ui/storage/RDDPage.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 6 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/resources/HistoryServerExpectations/executor_memory_usage_expectation.json 139 additions, 0 deletions...ServerExpectations/executor_memory_usage_expectation.json
- core/src/test/resources/HistoryServerExpectations/executor_node_blacklisting_expectation.json 31 additions, 10 deletions...rExpectations/executor_node_blacklisting_expectation.json
- core/src/test/resources/spark-events/app-20161116163331-0000 5 additions, 5 deletionscore/src/test/resources/spark-events/app-20161116163331-0000
- core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala 2 additions, 1 deletion.../org/apache/spark/deploy/history/HistoryServerSuite.scala
Loading
Please register or sign in to comment