-
- Downloads
[SPARK-6752] [STREAMING] [REVISED] Allow StreamingContext to be recreated from...
[SPARK-6752] [STREAMING] [REVISED] Allow StreamingContext to be recreated from checkpoint and existing SparkContext This is a revision of the earlier version (see #5773) that passed the active SparkContext explicitly through a new set of Java and Scala API. The drawbacks are. * Hard to implement in python. * New API introduced. This is even more confusing since we are introducing getActiveOrCreate in SPARK-7553 Furthermore, there is now a direct way get an existing active SparkContext or create a new on - SparkContext.getOrCreate(conf). Its better to use this to get the SparkContext rather than have a new API to explicitly pass the context. So in this PR I have * Removed the new versions of StreamingContext.getOrCreate() which took SparkContext * Added the ability to pick up existing SparkContext when the StreamingContext tries to create a SparkContext. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #6096 from tdas/SPARK-6752 and squashes the following commits: 53f4b2d [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-6752 f024b77 [Tathagata Das] Removed extra API and used SparkContext.getOrCreate
Showing
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 1 addition, 48 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala 0 additions, 45 deletions...pache/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java 3 additions, 22 deletions...rc/test/java/org/apache/spark/streaming/JavaAPISuite.java
- streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala 5 additions, 65 deletions...la/org/apache/spark/streaming/StreamingContextSuite.scala
Loading
Please register or sign in to comment