Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    c246b95d
    [SPARK-4841] fix zip with textFile() · c246b95d
    Davies Liu authored
    UTF8Deserializer can not be used in BatchedSerializer, so always use PickleSerializer() when change batchSize in zip().
    
    Also, if two RDD have the same batch size already, they did not need re-serialize any more.
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #3706 from davies/fix_4841 and squashes the following commits:
    
    20ce3a3 [Davies Liu] fix bug in _reserialize()
    e3ebf7c [Davies Liu] add comment
    379d2c8 [Davies Liu] fix zip with textFile()
    c246b95d
    History
    [SPARK-4841] fix zip with textFile()
    Davies Liu authored
    UTF8Deserializer can not be used in BatchedSerializer, so always use PickleSerializer() when change batchSize in zip().
    
    Also, if two RDD have the same batch size already, they did not need re-serialize any more.
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #3706 from davies/fix_4841 and squashes the following commits:
    
    20ce3a3 [Davies Liu] fix bug in _reserialize()
    e3ebf7c [Davies Liu] add comment
    379d2c8 [Davies Liu] fix zip with textFile()