Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    e77fa81a
    [SPARK-3554] [PySpark] use broadcast automatically for large closure · e77fa81a
    Davies Liu authored
    Py4j can not handle large string efficiently, so we should use broadcast for large closure automatically. (Broadcast use local filesystem to pass through data).
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2417 from davies/command and squashes the following commits:
    
    fbf4e97 [Davies Liu] bugfix
    aefd508 [Davies Liu] use broadcast automatically for large closure
    e77fa81a
    History
    [SPARK-3554] [PySpark] use broadcast automatically for large closure
    Davies Liu authored
    Py4j can not handle large string efficiently, so we should use broadcast for large closure automatically. (Broadcast use local filesystem to pass through data).
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2417 from davies/command and squashes the following commits:
    
    fbf4e97 [Davies Liu] bugfix
    aefd508 [Davies Liu] use broadcast automatically for large closure
worker.py 4.52 KiB