-
- Downloads
[SPARK-11359][STREAMING][KINESIS] Checkpoint to DynamoDB even when new data doesn't come in
Currently, the checkpoints to DynamoDB occur only when new data comes in, as we update the clock for the checkpointState. This PR makes the checkpoint a scheduled execution based on the `checkpointInterval`. Author: Burak Yavuz <brkyvz@gmail.com> Closes #9421 from brkyvz/kinesis-checkpoint.
Showing
- extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointState.scala 0 additions, 54 deletions...ache/spark/streaming/kinesis/KinesisCheckpointState.scala
- extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala 133 additions, 0 deletions.../apache/spark/streaming/kinesis/KinesisCheckpointer.scala
- extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala 33 additions, 5 deletions.../org/apache/spark/streaming/kinesis/KinesisReceiver.scala
- extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala 12 additions, 47 deletions...ache/spark/streaming/kinesis/KinesisRecordProcessor.scala
- extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointerSuite.scala 152 additions, 0 deletions...he/spark/streaming/kinesis/KinesisCheckpointerSuite.scala
- extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala 19 additions, 77 deletions...apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
Loading
Please register or sign in to comment