-
- Downloads
[SPARK-20863] Add metrics/instrumentation to LiveListenerBus
## What changes were proposed in this pull request? This patch adds Coda Hale metrics for instrumenting the `LiveListenerBus` in order to track the number of events received, dropped, and processed. In addition, it adds per-SparkListener-subclass timers to track message processing time. This is useful for identifying when slow third-party SparkListeners cause performance bottlenecks. See the new `LiveListenerBusMetrics` for a complete description of the new metrics. ## How was this patch tested? New tests in SparkListenerSuite, including a test to ensure proper counting of dropped listener events. Author: Josh Rosen <joshrosen@databricks.com> Closes #18083 from JoshRosen/listener-bus-metrics.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 5 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/internal/config/package.scala 6 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala 94 additions, 7 deletions...in/scala/org/apache/spark/scheduler/LiveListenerBus.scala
- core/src/main/scala/org/apache/spark/util/ListenerBus.scala 28 additions, 5 deletionscore/src/main/scala/org/apache/spark/util/ListenerBus.scala
- core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala 5 additions, 2 deletions...rg/apache/spark/scheduler/EventLoggingListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala 77 additions, 16 deletions...scala/org/apache/spark/scheduler/SparkListenerSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala 1 addition, 1 deletion...g/apache/spark/storage/BlockManagerReplicationSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 1 addition, 1 deletion...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- core/src/test/scala/org/apache/spark/ui/storage/StorageTabSuite.scala 2 additions, 3 deletions...t/scala/org/apache/spark/ui/storage/StorageTabSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala 1 addition, 3 deletions...rg/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
Loading
Please register or sign in to comment