Skip to content
  • Nicholas Chammas's avatar
    d14df06c
    [SPARK-6191] [EC2] Generalize ability to download libs · d14df06c
    Nicholas Chammas authored
    Right now we have a method to specifically download boto. This PR generalizes it so it's easy to download additional libraries if we want.
    
    For example, adding new external libraries for spark-ec2 is now as simple as:
    
    ```python
    external_libs = [
        {
             "name": "boto",
             "version": "2.34.0",
             "md5": "5556223d2d0cc4d06dd4829e671dcecd"
        },
        {
            "name": "PyYAML",
            "version": "3.11",
            "md5": "f50e08ef0fe55178479d3a618efe21db"
        },
        {
            "name": "argparse",
            "version": "1.3.0",
            "md5": "9bcf7f612190885c8c85e30ba41db3c7"
        }
    ]
    ```
    Likely use cases:
    * Downloading PyYAML to allow spark-ec2 configs to be persisted as a YAML file. ([SPARK-925](https://issues.apache.org/jira/browse/SPARK-925))
    * Downloading argparse to clean up / modernize our option parsing.
    
    First run output, with PyYAML and argparse added just for demonstration purposes:
    
    ```shell
    $ ./spark-ec2 --version
    Downloading external libraries that spark-ec2 needs from PyPI to /path/to/spark/ec2/lib...
    This should be a one-time operation.
     - Downloading boto...
     - Finished downloading boto.
     - Downloading PyYAML...
     - Finished downloading PyYAML.
     - Downloading argparse...
     - Finished downloading argparse.
    spark-ec2 1.2.1
    ```
    
    Output thereafter:
    
    ```shell
    $ ./spark-ec2 --version
    spark-ec2 1.2.1
    ```
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    
    Closes #4919 from nchammas/setup-ec2-libs and squashes the following commits:
    
    a077955 [Nicholas Chammas] print default region
    c95fb7d [Nicholas Chammas] to docstring
    5448845 [Nicholas Chammas] remove libs added for demo purposes
    60d8c23 [Nicholas Chammas] generalize ability to download libs
    d14df06c
    [SPARK-6191] [EC2] Generalize ability to download libs
    Nicholas Chammas authored
    Right now we have a method to specifically download boto. This PR generalizes it so it's easy to download additional libraries if we want.
    
    For example, adding new external libraries for spark-ec2 is now as simple as:
    
    ```python
    external_libs = [
        {
             "name": "boto",
             "version": "2.34.0",
             "md5": "5556223d2d0cc4d06dd4829e671dcecd"
        },
        {
            "name": "PyYAML",
            "version": "3.11",
            "md5": "f50e08ef0fe55178479d3a618efe21db"
        },
        {
            "name": "argparse",
            "version": "1.3.0",
            "md5": "9bcf7f612190885c8c85e30ba41db3c7"
        }
    ]
    ```
    Likely use cases:
    * Downloading PyYAML to allow spark-ec2 configs to be persisted as a YAML file. ([SPARK-925](https://issues.apache.org/jira/browse/SPARK-925))
    * Downloading argparse to clean up / modernize our option parsing.
    
    First run output, with PyYAML and argparse added just for demonstration purposes:
    
    ```shell
    $ ./spark-ec2 --version
    Downloading external libraries that spark-ec2 needs from PyPI to /path/to/spark/ec2/lib...
    This should be a one-time operation.
     - Downloading boto...
     - Finished downloading boto.
     - Downloading PyYAML...
     - Finished downloading PyYAML.
     - Downloading argparse...
     - Finished downloading argparse.
    spark-ec2 1.2.1
    ```
    
    Output thereafter:
    
    ```shell
    $ ./spark-ec2 --version
    spark-ec2 1.2.1
    ```
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    
    Closes #4919 from nchammas/setup-ec2-libs and squashes the following commits:
    
    a077955 [Nicholas Chammas] print default region
    c95fb7d [Nicholas Chammas] to docstring
    5448845 [Nicholas Chammas] remove libs added for demo purposes
    60d8c23 [Nicholas Chammas] generalize ability to download libs
Loading