-
- Downloads
[SPARK-21409][SS] Expose state store memory usage in SQL metrics and progress updates
## What changes were proposed in this pull request? Currently, there is no tracking of memory usage of state stores. This JIRA is to expose that through SQL metrics and StreamingQueryProgress. Additionally, added the ability to expose implementation-specific metrics through the StateStore APIs to the SQLMetrics. ## How was this patch tested? Added unit tests. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #18629 from tdas/SPARK-21409.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala 1 addition, 1 deletion.../sql/execution/streaming/FlatMapGroupsWithStateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala 4 additions, 12 deletions...ache/spark/sql/execution/streaming/ProgressReporter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala 8 additions, 2 deletions...cution/streaming/state/HDFSBackedStateStoreProvider.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala 40 additions, 8 deletions...ache/spark/sql/execution/streaming/state/StateStore.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala 43 additions, 9 deletions...che/spark/sql/execution/streaming/statefulOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala 10 additions, 2 deletions.../main/scala/org/apache/spark/sql/streaming/progress.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala 16 additions, 6 deletions...spark/sql/execution/streaming/state/StateStoreSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala 2 additions, 2 deletions...che/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala 26 additions, 19 deletions.../sql/streaming/StreamingQueryStatusAndProgressSuite.scala
Loading
Please register or sign in to comment