-
- Downloads
[SPARK-8701] [STREAMING] [WEBUI] Add input metadata in the batch page
This PR adds `metadata` to `InputInfo`. `InputDStream` can report its metadata for a batch and it will be shown in the batch page. For example,  FileInputDStream will display the new files for a batch, and DirectKafkaInputDStream will display its offset ranges. Author: zsxwing <zsxwing@gmail.com> Closes #7081 from zsxwing/input-metadata and squashes the following commits: f7abd9b [zsxwing] Revert the space changes in project/MimaExcludes.scala d906209 [zsxwing] Merge branch 'master' into input-metadata 74762da [zsxwing] Fix MiMa tests 7903e33 [zsxwing] Merge branch 'master' into input-metadata 450a46c [zsxwing] Address comments 1d94582 [zsxwing] Raname InputInfo to StreamInputInfo and change "metadata" to Map[String, Any] d496ae9 [zsxwing] Add input metadata in the batch page
Showing
- external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala 19 additions, 4 deletions...pache/spark/streaming/kafka/DirectKafkaInputDStream.scala
- external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala 1 addition, 1 deletion.../scala/org/apache/spark/streaming/kafka/OffsetRange.scala
- project/MimaExcludes.scala 6 additions, 0 deletionsproject/MimaExcludes.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala 9 additions, 1 deletion...org/apache/spark/streaming/dstream/FileInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala 2 additions, 2 deletions...apache/spark/streaming/dstream/ReceiverInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala 6 additions, 3 deletions...cala/org/apache/spark/streaming/scheduler/BatchInfo.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/InputInfoTracker.scala 31 additions, 7 deletions...g/apache/spark/streaming/scheduler/InputInfoTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala 1 addition, 2 deletions...a/org/apache/spark/streaming/scheduler/JobGenerator.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobSet.scala 2 additions, 2 deletions...n/scala/org/apache/spark/streaming/scheduler/JobSet.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala 40 additions, 3 deletions.../main/scala/org/apache/spark/streaming/ui/BatchPage.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/BatchUIData.scala 4 additions, 4 deletions...ain/scala/org/apache/spark/streaming/ui/BatchUIData.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala 3 additions, 2 deletions...che/spark/streaming/ui/StreamingJobProgressListener.scala
- streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala 3 additions, 3 deletions...a/org/apache/spark/streaming/StreamingListenerSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala 1 addition, 1 deletion...test/scala/org/apache/spark/streaming/TestSuiteBase.scala
- streaming/src/test/scala/org/apache/spark/streaming/scheduler/InputInfoTrackerSuite.scala 4 additions, 4 deletions...che/spark/streaming/scheduler/InputInfoTrackerSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/ui/StreamingJobProgressListenerSuite.scala 16 additions, 12 deletions...park/streaming/ui/StreamingJobProgressListenerSuite.scala
Loading
Please register or sign in to comment