-
- Downloads
[SPARK-18020][STREAMING][KINESIS] Checkpoint SHARD_END to finish reading closed shards
## What changes were proposed in this pull request? This pr is to fix an issue occurred when resharding Kinesis streams; the resharding makes the KCL throw an exception because Spark does not checkpoint `SHARD_END` when finishing reading closed shards in `KinesisRecordProcessor#shutdown`. This bug finally leads to stopping subscribing new split (or merged) shards. ## How was this patch tested? Added a test in `KinesisStreamSuite` to check if it works well when splitting/merging shards. Author: Takeshi YAMAMURO <linguin.m.s@gmail.com> Closes #16213 from maropu/SPARK-18020.
Showing
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala 14 additions, 1 deletion.../apache/spark/streaming/kinesis/KinesisCheckpointer.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala 27 additions, 3 deletions...org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KPLBasedKinesisTestUtils.scala 2 additions, 1 deletion...he/spark/streaming/kinesis/KPLBasedKinesisTestUtils.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointerSuite.scala 3 additions, 2 deletions...he/spark/streaming/kinesis/KinesisCheckpointerSuite.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala 70 additions, 0 deletions...g/apache/spark/streaming/kinesis/KinesisStreamSuite.scala
- python/pyspark/streaming/tests.py 1 addition, 1 deletionpython/pyspark/streaming/tests.py
Loading
Please register or sign in to comment