-
- Downloads
[SPARK-19405][STREAMING] Support for cross-account Kinesis reads via STS
- Add dependency on aws-java-sdk-sts - Replace SerializableAWSCredentials with new SerializableCredentialsProvider interface - Make KinesisReceiver take SerializableCredentialsProvider as argument and pass credential provider to KCL - Add new implementations of KinesisUtils.createStream() that take STS arguments - Make JavaKinesisStreamSuite test the entire KinesisUtils Java API - Update KCL/AWS SDK dependencies to 1.7.x/1.11.x ## What changes were proposed in this pull request? [JIRA link with detailed description.](https://issues.apache.org/jira/browse/SPARK-19405) * Replace SerializableAWSCredentials with new SerializableKCLAuthProvider class that takes 5 optional config params for configuring AWS auth and returns the appropriate credential provider object * Add new public createStream() APIs for specifying these parameters in KinesisUtils ## How was this patch tested? * Manually tested using explicit keypair and instance profile to read data from Kinesis stream in separate account (difficult to write a test orchestrating creation and assumption of IAM roles across separate accounts) * Expanded JavaKinesisStreamSuite to test the entire Java API in KinesisUtils ## License acknowledgement This contribution is my original work and that I license the work to the project under the project’s open source license. Author: Budde <budde@amazon.com> Closes #16744 from budde/master.
Showing
- external/kinesis-asl/pom.xml 5 additions, 0 deletionsexternal/kinesis-asl/pom.xml
- external/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java 1 addition, 1 deletion...che/spark/examples/streaming/JavaKinesisWordCountASL.java
- external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisExampleUtils.scala 35 additions, 0 deletions...apache/spark/examples/streaming/KinesisExampleUtils.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala 1 addition, 1 deletion...apache/spark/examples/streaming/KinesisWordCountASL.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala 3 additions, 5 deletions...pache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala 1 addition, 1 deletion.../apache/spark/streaming/kinesis/KinesisCheckpointer.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala 4 additions, 3 deletions.../apache/spark/streaming/kinesis/KinesisInputDStream.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala 13 additions, 38 deletions.../org/apache/spark/streaming/kinesis/KinesisReceiver.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala 1 addition, 1 deletion...ache/spark/streaming/kinesis/KinesisRecordProcessor.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala 12 additions, 2 deletions...org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala 173 additions, 19 deletions...ala/org/apache/spark/streaming/kinesis/KinesisUtils.scala
- external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SerializableCredentialsProvider.scala 85 additions, 0 deletions...k/streaming/kinesis/SerializableCredentialsProvider.scala
- external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java 34 additions, 1 deletion...pache/spark/streaming/kinesis/JavaKinesisStreamSuite.java
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisReceiverSuite.scala 21 additions, 4 deletions...apache/spark/streaming/kinesis/KinesisReceiverSuite.scala
- external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala 5 additions, 4 deletions...g/apache/spark/streaming/kinesis/KinesisStreamSuite.scala
- pom.xml 3 additions, 1 deletionpom.xml
- python/pyspark/streaming/kinesis.py 10 additions, 2 deletionspython/pyspark/streaming/kinesis.py
Loading
Please register or sign in to comment