Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    2fc8aca0
    [SPARK-1065] [PySpark] improve supporting for large broadcast · 2fc8aca0
    Davies Liu authored
    Passing large object by py4j is very slow (cost much memory), so pass broadcast objects via files (similar to parallelize()).
    
    Add an option to keep object in driver (it's False by default) to save memory in driver.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1912 from davies/broadcast and squashes the following commits:
    
    e06df4a [Davies Liu] load broadcast from disk in driver automatically
    db3f232 [Davies Liu] fix serialization of accumulator
    631a827 [Davies Liu] Merge branch 'master' into broadcast
    c7baa8c [Davies Liu] compress serrialized broadcast and command
    9a7161f [Davies Liu] fix doc tests
    e93cf4b [Davies Liu] address comments: add test
    6226189 [Davies Liu] improve large broadcast
    2fc8aca0
    History
    [SPARK-1065] [PySpark] improve supporting for large broadcast
    Davies Liu authored
    Passing large object by py4j is very slow (cost much memory), so pass broadcast objects via files (similar to parallelize()).
    
    Add an option to keep object in driver (it's False by default) to save memory in driver.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1912 from davies/broadcast and squashes the following commits:
    
    e06df4a [Davies Liu] load broadcast from disk in driver automatically
    db3f232 [Davies Liu] fix serialization of accumulator
    631a827 [Davies Liu] Merge branch 'master' into broadcast
    c7baa8c [Davies Liu] compress serrialized broadcast and command
    9a7161f [Davies Liu] fix doc tests
    e93cf4b [Davies Liu] address comments: add test
    6226189 [Davies Liu] improve large broadcast
tests.py 42.21 KiB