Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    091d32c5
    [SPARK-3971] [MLLib] [PySpark] hotfix: Customized pickler should work in cluster mode · 091d32c5
    Davies Liu authored
    Customized pickler should be registered before unpickling, but in executor, there is no way to register the picklers before run the tasks.
    
    So, we need to register the picklers in the tasks itself, duplicate the javaToPython() and pythonToJava() in MLlib, call SerDe.initialize() before pickling or unpickling.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2830 from davies/fix_pickle and squashes the following commits:
    
    0c85fb9 [Davies Liu] revert the privacy change
    6b94e15 [Davies Liu] use JavaConverters instead of JavaConversions
    0f02050 [Davies Liu] hotfix: Customized pickler does not work in cluster
    091d32c5
    History
    [SPARK-3971] [MLLib] [PySpark] hotfix: Customized pickler should work in cluster mode
    Davies Liu authored
    Customized pickler should be registered before unpickling, but in executor, there is no way to register the picklers before run the tasks.
    
    So, we need to register the picklers in the tasks itself, duplicate the javaToPython() and pythonToJava() in MLlib, call SerDe.initialize() before pickling or unpickling.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2830 from davies/fix_pickle and squashes the following commits:
    
    0c85fb9 [Davies Liu] revert the privacy change
    6b94e15 [Davies Liu] use JavaConverters instead of JavaConversions
    0f02050 [Davies Liu] hotfix: Customized pickler does not work in cluster