-
- Downloads
[SPARK-19911][STREAMING] Add builder interface for Kinesis DStreams
## What changes were proposed in this pull request? - Add new KinesisDStream.scala containing KinesisDStream.Builder class - Add KinesisDStreamBuilderSuite test suite - Make KinesisInputDStream ctor args package private for testing - Add JavaKinesisDStreamBuilderSuite test suite - Add args to KinesisInputDStream and KinesisReceiver for optional service-specific auth (Kinesis, DynamoDB and CloudWatch) ## How was this patch tested? Added ```KinesisDStreamBuilderSuite``` to verify builder class works as expected Author: Adam Budde <budde@amazon.com> Closes #17250 from budde/KinesisStreamBuilder.
Showing
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala 3 additions, 3 deletions...pache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala 247 additions, 12 deletions.../apache/spark/streaming/kinesis/KinesisInputDStream.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala 15 additions, 5 deletions.../org/apache/spark/streaming/kinesis/KinesisReceiver.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala 23 additions, 20 deletions...ala/org/apache/spark/streaming/kinesis/KinesisUtils.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala 182 additions, 0 deletions.../apache/spark/streaming/kinesis/SparkAWSCredentials.scala
- external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java 63 additions, 0 deletions...treaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala 115 additions, 0 deletions...k/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala 0 additions, 23 deletions...apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala 1 addition, 1 deletion...g/apache/spark/streaming/kinesis/KinesisStreamSuite.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentialsBuilderSuite.scala 100 additions, 0 deletions...k/streaming/kinesis/SparkAWSCredentialsBuilderSuite.scala
Loading
Please register or sign in to comment