Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    abf588f4
    [SPARK-3749] [PySpark] fix bugs in broadcast large closure of RDD · abf588f4
    Davies Liu authored
    1. broadcast is triggle unexpected
    2. fd is leaked in JVM (also leak in parallelize())
    3. broadcast is not unpersisted in JVM after RDD is not be used any more.
    
    cc JoshRosen , sorry for these stupid bugs.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2603 from davies/fix_broadcast and squashes the following commits:
    
    080a743 [Davies Liu] fix bugs in broadcast large closure of RDD
    abf588f4
    History
    [SPARK-3749] [PySpark] fix bugs in broadcast large closure of RDD
    Davies Liu authored
    1. broadcast is triggle unexpected
    2. fd is leaked in JVM (also leak in parallelize())
    3. broadcast is not unpersisted in JVM after RDD is not be used any more.
    
    cc JoshRosen , sorry for these stupid bugs.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2603 from davies/fix_broadcast and squashes the following commits:
    
    080a743 [Davies Liu] fix bugs in broadcast large closure of RDD
tests.py 62.59 KiB