Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    bb96012b
    [SPARK-3679] [PySpark] pickle the exact globals of functions · bb96012b
    Davies Liu authored
    function.func_code.co_names has all the names used in the function, including name of attributes. It will pickle some unnecessary globals if there is a global having the same name with attribute (in co_names).
    
    There is a regression introduced by #2144, revert part of changes in that PR.
    
    cc JoshRosen
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2522 from davies/globals and squashes the following commits:
    
    dfbccf5 [Davies Liu] fix bug while pickle globals of function
    bb96012b
    History
    [SPARK-3679] [PySpark] pickle the exact globals of functions
    Davies Liu authored
    function.func_code.co_names has all the names used in the function, including name of attributes. It will pickle some unnecessary globals if there is a global having the same name with attribute (in co_names).
    
    There is a regression introduced by #2144, revert part of changes in that PR.
    
    cc JoshRosen
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2522 from davies/globals and squashes the following commits:
    
    dfbccf5 [Davies Liu] fix bug while pickle globals of function