Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    8ead999f
    [SPARK-5223] [MLlib] [PySpark] fix MapConverter and ListConverter in MLlib · 8ead999f
    Davies Liu authored
    It will introduce problems if the object in dict/list/tuple can not support by py4j, such as Vector.
    Also, pickle may have better performance for larger object (less RPC).
    
    In some cases that the object in dict/list can not be pickled (such as JavaObject), we should still use MapConvert/ListConvert.
    
    This PR should be ported into branch-1.2
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #4023 from davies/listconvert and squashes the following commits:
    
    55d4ab2 [Davies Liu] fix MapConverter and ListConverter in MLlib
    8ead999f
    History
    [SPARK-5223] [MLlib] [PySpark] fix MapConverter and ListConverter in MLlib
    Davies Liu authored
    It will introduce problems if the object in dict/list/tuple can not support by py4j, such as Vector.
    Also, pickle may have better performance for larger object (less RPC).
    
    In some cases that the object in dict/list can not be pickled (such as JavaObject), we should still use MapConvert/ListConvert.
    
    This PR should be ported into branch-1.2
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #4023 from davies/listconvert and squashes the following commits:
    
    55d4ab2 [Davies Liu] fix MapConverter and ListConverter in MLlib