Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    e4f42631
    [SPARK-3886] [PySpark] simplify serializer, use AutoBatchedSerializer by default. · e4f42631
    Davies Liu authored
    This PR simplify serializer, always use batched serializer (AutoBatchedSerializer as default), even batch size is 1.
    
    Author: Davies Liu <davies@databricks.com>
    
    This patch had conflicts when merged, resolved by
    Committer: Josh Rosen <joshrosen@databricks.com>
    
    Closes #2920 from davies/fix_autobatch and squashes the following commits:
    
    e544ef9 [Davies Liu] revert unrelated change
    6880b14 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    1d557fc [Davies Liu] fix tests
    8180907 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    76abdce [Davies Liu] clean up
    53fa60b [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    d7ac751 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    2cc2497 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    b4292ce [Davies Liu] fix bug in master
    d79744c [Davies Liu] recover hive tests
    be37ece [Davies Liu] refactor
    eb3938d [Davies Liu] refactor serializer in scala
    8d77ef2 [Davies Liu] simplify serializer, use AutoBatchedSerializer by default.
    e4f42631
    History
    [SPARK-3886] [PySpark] simplify serializer, use AutoBatchedSerializer by default.
    Davies Liu authored
    This PR simplify serializer, always use batched serializer (AutoBatchedSerializer as default), even batch size is 1.
    
    Author: Davies Liu <davies@databricks.com>
    
    This patch had conflicts when merged, resolved by
    Committer: Josh Rosen <joshrosen@databricks.com>
    
    Closes #2920 from davies/fix_autobatch and squashes the following commits:
    
    e544ef9 [Davies Liu] revert unrelated change
    6880b14 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    1d557fc [Davies Liu] fix tests
    8180907 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    76abdce [Davies Liu] clean up
    53fa60b [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    d7ac751 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    2cc2497 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_autobatch
    b4292ce [Davies Liu] fix bug in master
    d79744c [Davies Liu] recover hive tests
    be37ece [Davies Liu] refactor
    eb3938d [Davies Liu] refactor serializer in scala
    8d77ef2 [Davies Liu] simplify serializer, use AutoBatchedSerializer by default.
common.py 4.00 KiB