Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    872538c6
    [SPARK-2494] [PySpark] make hash of None consistant cross machines · 872538c6
    Davies Liu authored
    In CPython, hash of None is different cross machines, it will cause wrong result during shuffle. This PR will fix this.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1371 from davies/hash_of_none and squashes the following commits:
    
    d01745f [Davies Liu] add comments, remove outdated unit tests
    5467141 [Davies Liu] disable hijack of hash, use it only for partitionBy()
    b7118aa [Davies Liu] use __builtin__ instead of __builtins__
    839e417 [Davies Liu] hijack hash to make hash of None consistant cross machines
    872538c6
    History
    [SPARK-2494] [PySpark] make hash of None consistant cross machines
    Davies Liu authored
    In CPython, hash of None is different cross machines, it will cause wrong result during shuffle. This PR will fix this.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1371 from davies/hash_of_none and squashes the following commits:
    
    d01745f [Davies Liu] add comments, remove outdated unit tests
    5467141 [Davies Liu] disable hijack of hash, use it only for partitionBy()
    b7118aa [Davies Liu] use __builtin__ instead of __builtins__
    839e417 [Davies Liu] hijack hash to make hash of None consistant cross machines