-
- Downloads
[SPARK-6752][Streaming] Allow StreamingContext to be recreated from checkpoint...
[SPARK-6752][Streaming] Allow StreamingContext to be recreated from checkpoint and existing SparkContext Currently if you want to create a StreamingContext from checkpoint information, the system will create a new SparkContext. This prevent StreamingContext to be recreated from checkpoints in managed environments where SparkContext is precreated. The solution in this PR: Introduce the following methods on StreamingContext 1. `new StreamingContext(checkpointDirectory, sparkContext)` Recreate StreamingContext from checkpoint using the provided SparkContext 2. `StreamingContext.getOrCreate(checkpointDirectory, sparkContext, createFunction: SparkContext => StreamingContext)` If checkpoint file exists, then recreate StreamingContext using the provided SparkContext (that is, call 1.), else create StreamingContext using the provided createFunction TODO: the corresponding Java and Python API has to be added as well. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #5428 from tdas/SPARK-6752 and squashes the following commits: 94db63c [Tathagata Das] Fix long line. 524f519 [Tathagata Das] Many changes based on PR comments. eabd092 [Tathagata Das] Added Function0, Java API and unit tests for StreamingContext.getOrCreate 36a7823 [Tathagata Das] Minor changes. 204814e [Tathagata Das] Added StreamingContext.getOrCreate with existing SparkContext
Showing
- core/src/main/java/org/apache/spark/api/java/function/Function0.java 27 additions, 0 deletions...in/java/org/apache/spark/api/java/function/Function0.java
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 20 additions, 6 deletions...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 71 additions, 14 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala 117 additions, 2 deletions...pache/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java 108 additions, 37 deletions...rc/test/java/org/apache/spark/streaming/JavaAPISuite.java
- streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala 1 addition, 2 deletions...st/scala/org/apache/spark/streaming/CheckpointSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala 159 additions, 0 deletions...la/org/apache/spark/streaming/StreamingContextSuite.scala
Loading
Please register or sign in to comment