-
- Downloads
Many changes to ensure better 2nd recovery if 2nd failure happens while
recovering from 1st failure - Made the scheduler to checkpoint after clearing old metadata which ensures that a new checkpoint is written as soon as at least one batch gets computed while recovering from a failure. This ensures that if there is a 2nd failure while recovering from 1st failure, the system start 2nd recovery from a newer checkpoint. - Modified Checkpoint writer to write checkpoint in a different thread. - Added a check to make sure that compute for InputDStreams gets called only for strictly increasing times. - Changed implementation of slice to call getOrCompute on parent DStream in time-increasing order. - Added testcase to test slice. - Fixed testGroupByKeyAndWindow testcase in JavaAPISuite to verify results with expected output in an order-independent manner.
Showing
- streaming/src/main/scala/spark/streaming/Checkpoint.scala 44 additions, 23 deletionsstreaming/src/main/scala/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/spark/streaming/DStream.scala 17 additions, 10 deletionsstreaming/src/main/scala/spark/streaming/DStream.scala
- streaming/src/main/scala/spark/streaming/DStreamGraph.scala 10 additions, 3 deletionsstreaming/src/main/scala/spark/streaming/DStreamGraph.scala
- streaming/src/main/scala/spark/streaming/JobManager.scala 6 additions, 2 deletionsstreaming/src/main/scala/spark/streaming/JobManager.scala
- streaming/src/main/scala/spark/streaming/Scheduler.scala 16 additions, 11 deletionsstreaming/src/main/scala/spark/streaming/Scheduler.scala
- streaming/src/main/scala/spark/streaming/StreamingContext.scala 2 additions, 5 deletions...ing/src/main/scala/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/spark/streaming/Time.scala 5 additions, 6 deletionsstreaming/src/main/scala/spark/streaming/Time.scala
- streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala 3 additions, 4 deletions...scala/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/main/scala/spark/streaming/dstream/InputDStream.scala 35 additions, 1 deletion...src/main/scala/spark/streaming/dstream/InputDStream.scala
- streaming/src/main/scala/spark/streaming/dstream/TwitterInputDStream.scala 2 additions, 2 deletions...n/scala/spark/streaming/dstream/TwitterInputDStream.scala
- streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala 1 addition, 1 deletion...c/main/scala/spark/streaming/util/MasterFailureTest.scala
- streaming/src/test/java/spark/streaming/JavaAPISuite.java 38 additions, 16 deletionsstreaming/src/test/java/spark/streaming/JavaAPISuite.java
- streaming/src/test/java/spark/streaming/JavaTestUtils.scala 1 addition, 0 deletionsstreaming/src/test/java/spark/streaming/JavaTestUtils.scala
- streaming/src/test/resources/log4j.properties 1 addition, 3 deletionsstreaming/src/test/resources/log4j.properties
- streaming/src/test/scala/spark/streaming/BasicOperationsSuite.scala 20 additions, 0 deletions...src/test/scala/spark/streaming/BasicOperationsSuite.scala
- streaming/src/test/scala/spark/streaming/CheckpointSuite.scala 1 addition, 4 deletions...ming/src/test/scala/spark/streaming/CheckpointSuite.scala
- streaming/src/test/scala/spark/streaming/TestSuiteBase.scala 2 additions, 5 deletionsstreaming/src/test/scala/spark/streaming/TestSuiteBase.scala
- streaming/src/test/scala/spark/streaming/WindowOperationsSuite.scala 4 additions, 1 deletion...rc/test/scala/spark/streaming/WindowOperationsSuite.scala
Loading
Please register or sign in to comment