Skip to content
Snippets Groups Projects
  • Dan Osipov's avatar
    b2017126
    [SPARK-787] Add S3 configuration parameters to the EC2 deploy scripts · b2017126
    Dan Osipov authored
    When deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't.
    
    This PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58
    
    Author: Dan Osipov <daniil.osipov@shazam.com>
    
    Closes #1120 from danosipov/s3_credentials and squashes the following commits:
    
    758da8b [Dan Osipov] Modify documentation to include the new parameter
    71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment
    7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance
    39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts
    b2017126
    History
    [SPARK-787] Add S3 configuration parameters to the EC2 deploy scripts
    Dan Osipov authored
    When deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't.
    
    This PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58
    
    Author: Dan Osipov <daniil.osipov@shazam.com>
    
    Closes #1120 from danosipov/s3_credentials and squashes the following commits:
    
    758da8b [Dan Osipov] Modify documentation to include the new parameter
    71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment
    7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance
    39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts