-
- Downloads
[SPARK-6077] Remove streaming tab while stopping StreamingContext
Currently we would create a new streaming tab for each streamingContext even if there's already one on the same sparkContext which would cause duplicate StreamingTab created and none of them is taking effect. snapshot: https://www.dropbox.com/s/t4gd6hqyqo0nivz/bad%20multiple%20streamings.png?dl=0 How to reproduce: 1) import org.apache.spark.SparkConf import org.apache.spark.streaming. {Seconds, StreamingContext} import org.apache.spark.storage.StorageLevel val ssc = new StreamingContext(sc, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER) val words = lines.flatMap(_.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() ssc.start() ..... 2) ssc.stop(false) val ssc = new StreamingContext(sc, Seconds(1)) val lines = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER) val words = lines.flatMap(_.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() ssc.start() Author: lisurprise <zhichao.li@intel.com> Closes #4828 from zhichao-li/master and squashes the following commits: c329806 [lisurprise] add test for attaching/detaching streaming tab 51e6c7f [lisurprise] move detach method into StreamingTab 31a44fa [lisurprise] add unit test for attaching and detaching new tab db25ed2 [lisurprise] clean code 8281bcb [lisurprise] clean code 193c542 [lisurprise] remove streaming tab while closing streaming context
Showing
- core/src/main/scala/org/apache/spark/ui/WebUI.scala 23 additions, 5 deletionscore/src/main/scala/org/apache/spark/ui/WebUI.scala
- core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala 48 additions, 2 deletions.../src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
- core/src/test/scala/org/apache/spark/ui/UISuite.scala 1 addition, 37 deletionscore/src/test/scala/org/apache/spark/ui/UISuite.scala
- streaming/pom.xml 5 additions, 0 deletionsstreaming/pom.xml
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 1 addition, 0 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala 2 additions, 2 deletions...n/scala/org/apache/spark/streaming/ui/StreamingPage.scala
- streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingTab.scala 4 additions, 0 deletions...in/scala/org/apache/spark/streaming/ui/StreamingTab.scala
- streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala 95 additions, 0 deletions...st/scala/org/apache/spark/streaming/UISeleniumSuite.scala
Loading
Please register or sign in to comment