-
- Downloads
[SPARK-4608][Streaming] Reorganize StreamingContext implicit to improve API convenience
There is only one implicit function `toPairDStreamFunctions` in `StreamingContext`. This PR did similar reorganization like [SPARK-4397](https://issues.apache.org/jira/browse/SPARK-4397). Compiled the following codes with Spark Streaming 1.1.0 and ran it with this PR. Everything is fine. ```Scala import org.apache.spark._ import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ object StreamingApp { def main(args: Array[String]) { val conf = new SparkConf().setMaster("local[2]").setAppName("FileWordCount") val ssc = new StreamingContext(conf, Seconds(10)) val lines = ssc.textFileStream("/some/path") val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination() } } ``` Author: zsxwing <zsxwing@gmail.com> Closes #3464 from zsxwing/SPARK-4608 and squashes the following commits: aa6d44a [zsxwing] Fix a copy-paste error f74c190 [zsxwing] Merge branch 'master' into SPARK-4608 e6f9cc9 [zsxwing] Update the docs 27833bb [zsxwing] Remove `import StreamingContext._` c15162c [zsxwing] Reorganize StreamingContext implicit to improve API convenience
Showing
- docs/streaming-programming-guide.md 2 additions, 2 deletionsdocs/streaming-programming-guide.md
- examples/scala-2.10/src/main/scala/org/apache/spark/examples/streaming/KafkaWordCount.scala 0 additions, 1 deletion.../org/apache/spark/examples/streaming/KafkaWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala 0 additions, 1 deletion.../org/apache/spark/examples/streaming/CustomReceiver.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala 0 additions, 1 deletion...a/org/apache/spark/examples/streaming/HdfsWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/MQTTWordCount.scala 0 additions, 1 deletion...a/org/apache/spark/examples/streaming/MQTTWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala 0 additions, 1 deletion...rg/apache/spark/examples/streaming/NetworkWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/QueueStream.scala 0 additions, 1 deletion...ala/org/apache/spark/examples/streaming/QueueStream.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala 0 additions, 1 deletion...park/examples/streaming/RecoverableNetworkWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala 0 additions, 1 deletion...e/spark/examples/streaming/StatefulNetworkWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/TwitterAlgebirdCMS.scala 0 additions, 1 deletion.../apache/spark/examples/streaming/TwitterAlgebirdCMS.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/TwitterPopularTags.scala 0 additions, 1 deletion.../apache/spark/examples/streaming/TwitterPopularTags.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/ZeroMQWordCount.scala 0 additions, 1 deletion...org/apache/spark/examples/streaming/ZeroMQWordCount.scala
- examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewStream.scala 0 additions, 1 deletion...spark/examples/streaming/clickstream/PageViewStream.scala
- mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala 0 additions, 1 deletion...a/org/apache/spark/mllib/clustering/StreamingKMeans.scala
- mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala 0 additions, 1 deletion...che/spark/mllib/regression/StreamingLinearAlgorithm.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 4 additions, 3 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala 1 addition, 2 deletions...org/apache/spark/streaming/api/java/JavaPairDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala 16 additions, 6 deletions...in/scala/org/apache/spark/streaming/dstream/DStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala 1 addition, 3 deletions...apache/spark/streaming/dstream/PairDStreamFunctions.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/ReducedWindowedDStream.scala 0 additions, 2 deletions...ache/spark/streaming/dstream/ReducedWindowedDStream.scala
Loading
Please register or sign in to comment