-
- Downloads
[SPARK-17812][SQL][KAFKA] Assign and specific startingOffsets for structured stream
## What changes were proposed in this pull request? startingOffsets takes specific per-topicpartition offsets as a json argument, usable with any consumer strategy assign with specific topicpartitions as a consumer strategy ## How was this patch tested? Unit tests Author: cody koeninger <cody@koeninger.org> Closes #15504 from koeninger/SPARK-17812.
Showing
- docs/structured-streaming-kafka-integration.md 26 additions, 12 deletionsdocs/structured-streaming-kafka-integration.md
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala 93 additions, 0 deletions.../main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala 57 additions, 7 deletions...ain/scala/org/apache/spark/sql/kafka010/KafkaSource.scala
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala 26 additions, 26 deletions...a/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/StartingOffsets.scala 32 additions, 0 deletions...scala/org/apache/spark/sql/kafka010/StartingOffsets.scala
- external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/JsonUtilsSuite.scala 45 additions, 0 deletions.../scala/org/apache/spark/sql/kafka010/JsonUtilsSuite.scala
- external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala 103 additions, 11 deletions...cala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala
- external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala 13 additions, 1 deletion.../scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
Loading
Please register or sign in to comment