-
- Downloads
Merge pull request #394 from tdas/error-handling
Better error handling in Spark Streaming and more API cleanup Earlier errors in jobs generated by Spark Streaming (or in the generation of jobs) could not be caught from the main driver thread (i.e. the thread that called StreamingContext.start()) as it would be thrown in different threads. With this change, after `ssc.start`, one can call `ssc.awaitTermination()` which will be block until the ssc is closed, or there is an exception. This makes it easier to debug. This change also adds ssc.stop(<stop-spark-context>) where you can stop StreamingContext without stopping the SparkContext. Also fixes the bug that came up with PRs #393 and #381. MetadataCleaner default value has been changed from 3500 to -1 for normal SparkContext and 3600 when creating a StreamingContext. Also, updated StreamingListenerBus with changes similar to SparkListenerBus in #392. And changed a lot of protected[streaming] to private[streaming].
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/util/MetadataCleaner.scala
- external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala 1 addition, 1 deletion...ala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala
- external/twitter/src/test/scala/org/apache/spark/streaming/twitter/TwitterStreamSuite.scala 1 addition, 1 deletion...g/apache/spark/streaming/twitter/TwitterStreamSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/ContextWaiter.scala 28 additions, 0 deletions...main/scala/org/apache/spark/streaming/ContextWaiter.scala
- streaming/src/main/scala/org/apache/spark/streaming/DStream.scala 31 additions, 31 deletions...g/src/main/scala/org/apache/spark/streaming/DStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala 7 additions, 1 deletion.../main/scala/org/apache/spark/streaming/DStreamGraph.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 50 additions, 62 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala 20 additions, 1 deletion...pache/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala 1 addition, 1 deletion...ala/org/apache/spark/streaming/dstream/InputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/NetworkInputDStream.scala 1 addition, 1 deletion.../apache/spark/streaming/dstream/NetworkInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/Job.scala 4 additions, 5 deletions...main/scala/org/apache/spark/streaming/scheduler/Job.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala 49 additions, 32 deletions...a/org/apache/spark/streaming/scheduler/JobGenerator.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala 97 additions, 44 deletions...a/org/apache/spark/streaming/scheduler/JobScheduler.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobSet.scala 9 additions, 9 deletions...n/scala/org/apache/spark/streaming/scheduler/JobSet.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/NetworkInputTracker.scala 27 additions, 13 deletions...pache/spark/streaming/scheduler/NetworkInputTracker.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListener.scala 2 additions, 1 deletion.../apache/spark/streaming/scheduler/StreamingListener.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListenerBus.scala 15 additions, 6 deletions...ache/spark/streaming/scheduler/StreamingListenerBus.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/RecurringTimer.scala 1 addition, 12 deletions...cala/org/apache/spark/streaming/util/RecurringTimer.scala
- streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala 1 addition, 5 deletions...ala/org/apache/spark/streaming/BasicOperationsSuite.scala
Loading
Please register or sign in to comment