Skip to content
Snippets Groups Projects
  • Adam Budde's avatar
    e4065376
    [SPARK-19405][STREAMING] Support for cross-account Kinesis reads via STS · e4065376
    Adam Budde authored
    - Add dependency on aws-java-sdk-sts
    - Replace SerializableAWSCredentials with new SerializableCredentialsProvider interface
    - Make KinesisReceiver take SerializableCredentialsProvider as argument and
      pass credential provider to KCL
    - Add new implementations of KinesisUtils.createStream() that take STS
      arguments
    - Make JavaKinesisStreamSuite test the entire KinesisUtils Java API
    - Update KCL/AWS SDK dependencies to 1.7.x/1.11.x
    
    ## What changes were proposed in this pull request?
    
    [JIRA link with detailed description.](https://issues.apache.org/jira/browse/SPARK-19405)
    
    * Replace SerializableAWSCredentials with new SerializableKCLAuthProvider class that takes 5 optional config params for configuring AWS auth and returns the appropriate credential provider object
    * Add new public createStream() APIs for specifying these parameters in KinesisUtils
    
    ## How was this patch tested?
    
    * Manually tested using explicit keypair and instance profile to read data from Kinesis stream in separate account (difficult to write a test orchestrating creation and assumption of IAM roles across separate accounts)
    * Expanded JavaKinesisStreamSuite to test the entire Java API in KinesisUtils
    
    ## License acknowledgement
    This contribution is my original work and that I license the work to the project under the project’s open source license.
    
    Author: Budde <budde@amazon.com>
    
    Closes #16744 from budde/master.
    e4065376
    History
    [SPARK-19405][STREAMING] Support for cross-account Kinesis reads via STS
    Adam Budde authored
    - Add dependency on aws-java-sdk-sts
    - Replace SerializableAWSCredentials with new SerializableCredentialsProvider interface
    - Make KinesisReceiver take SerializableCredentialsProvider as argument and
      pass credential provider to KCL
    - Add new implementations of KinesisUtils.createStream() that take STS
      arguments
    - Make JavaKinesisStreamSuite test the entire KinesisUtils Java API
    - Update KCL/AWS SDK dependencies to 1.7.x/1.11.x
    
    ## What changes were proposed in this pull request?
    
    [JIRA link with detailed description.](https://issues.apache.org/jira/browse/SPARK-19405)
    
    * Replace SerializableAWSCredentials with new SerializableKCLAuthProvider class that takes 5 optional config params for configuring AWS auth and returns the appropriate credential provider object
    * Add new public createStream() APIs for specifying these parameters in KinesisUtils
    
    ## How was this patch tested?
    
    * Manually tested using explicit keypair and instance profile to read data from Kinesis stream in separate account (difficult to write a test orchestrating creation and assumption of IAM roles across separate accounts)
    * Expanded JavaKinesisStreamSuite to test the entire Java API in KinesisUtils
    
    ## License acknowledgement
    This contribution is my original work and that I license the work to the project under the project’s open source license.
    
    Author: Budde <budde@amazon.com>
    
    Closes #16744 from budde/master.