Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    da33acb8
    [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6 · da33acb8
    Davies Liu authored
    Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.
    
    There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.
    
    I had send an PR to Pyrolite to fix it:  https://github.com/irmen/Pyrolite/pull/11
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2365 from davies/pickle and squashes the following commits:
    
    f44f771 [Davies Liu] enable tests about array
    3908f5c [Davies Liu] Merge branch 'master' into pickle
    c77c87b [Davies Liu] cleanup debugging code
    60e4e2f [Davies Liu] support unpickle array.array for Python 2.6
    da33acb8
    History
    [SPARK-2951] [PySpark] support unpickle array.array for Python 2.6
    Davies Liu authored
    Pyrolite can not unpickle array.array which pickled by Python 2.6, this patch fix it by extend Pyrolite.
    
    There is a bug in Pyrolite when unpickle array of float/double, this patch workaround it by reverse the endianness for float/double. This workaround should be removed after Pyrolite have a new release to fix this issue.
    
    I had send an PR to Pyrolite to fix it:  https://github.com/irmen/Pyrolite/pull/11
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2365 from davies/pickle and squashes the following commits:
    
    f44f771 [Davies Liu] enable tests about array
    3908f5c [Davies Liu] Merge branch 'master' into pickle
    c77c87b [Davies Liu] cleanup debugging code
    60e4e2f [Davies Liu] support unpickle array.array for Python 2.6