Skip to content
  • Josh Rosen's avatar
    c28083f4
    [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it · c28083f4
    Josh Rosen authored
    This patch upgrades `spark-ec2`'s Boto version to 2.34.0, since this is blocking several features.  Newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources.
    
    Therefore, this patch also changes spark-ec2 to automatically download Boto from PyPi if it's not present in `SPARK_EC2_DIR/lib`, similar to what we do in the `sbt/sbt` script. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster.  By performing the downloading in spark_ec2.py instead of the Bash script, this should also work for Windows users.
    
    I've tested this with Python 2.6, too.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #3737 from JoshRosen/update-boto and squashes the following commits:
    
    0aa43cc [Josh Rosen] Remove unused setup_standalone_cluster() method.
    f02935d [Josh Rosen] Enable Python deprecation warnings and fix one Boto warning:
    587ae89 [Josh Rosen] [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
    c28083f4
    [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
    Josh Rosen authored
    This patch upgrades `spark-ec2`'s Boto version to 2.34.0, since this is blocking several features.  Newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources.
    
    Therefore, this patch also changes spark-ec2 to automatically download Boto from PyPi if it's not present in `SPARK_EC2_DIR/lib`, similar to what we do in the `sbt/sbt` script. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster.  By performing the downloading in spark_ec2.py instead of the Bash script, this should also work for Windows users.
    
    I've tested this with Python 2.6, too.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #3737 from JoshRosen/update-boto and squashes the following commits:
    
    0aa43cc [Josh Rosen] Remove unused setup_standalone_cluster() method.
    f02935d [Josh Rosen] Enable Python deprecation warnings and fix one Boto warning:
    587ae89 [Josh Rosen] [SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
Loading