Skip to content
  • Simon Hafner's avatar
    83a621a5
    [SPARK-8821] [EC2] Switched to binary mode for file reading · 83a621a5
    Simon Hafner authored
    Otherwise the script will crash with
    
        - Downloading boto...
        Traceback (most recent call last):
          File "ec2/spark_ec2.py", line 148, in <module>
            setup_external_libs(external_libs)
          File "ec2/spark_ec2.py", line 128, in setup_external_libs
            if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
          File "/usr/lib/python3.4/codecs.py", line 319, in decode
            (result, consumed) = self._buffer_decode(data, self.errors, final)
        UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
    
    In case of an utf8 env setting.
    
    Author: Simon Hafner <hafnersimon@gmail.com>
    
    Closes #7215 from reactormonk/branch-1.4 and squashes the following commits:
    
    e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode
    83a621a5
    [SPARK-8821] [EC2] Switched to binary mode for file reading
    Simon Hafner authored
    Otherwise the script will crash with
    
        - Downloading boto...
        Traceback (most recent call last):
          File "ec2/spark_ec2.py", line 148, in <module>
            setup_external_libs(external_libs)
          File "ec2/spark_ec2.py", line 128, in setup_external_libs
            if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
          File "/usr/lib/python3.4/codecs.py", line 319, in decode
            (result, consumed) = self._buffer_decode(data, self.errors, final)
        UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
    
    In case of an utf8 env setting.
    
    Author: Simon Hafner <hafnersimon@gmail.com>
    
    Closes #7215 from reactormonk/branch-1.4 and squashes the following commits:
    
    e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode
Loading