Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    0d8cdf0e
    [SPARK-3681] [SQL] [PySpark] fix serialization of List and Map in SchemaRDD · 0d8cdf0e
    Davies Liu authored
    Currently, the schema of object in ArrayType or MapType is attached lazily, it will have better performance but introduce issues while serialization or accessing nested objects.
    
    This patch will apply schema to the objects of ArrayType or MapType immediately when accessing them, will be a little bit slower, but much robust.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2526 from davies/nested and squashes the following commits:
    
    2399ae5 [Davies Liu] fix serialization of List and Map in SchemaRDD
    0d8cdf0e
    History
    [SPARK-3681] [SQL] [PySpark] fix serialization of List and Map in SchemaRDD
    Davies Liu authored
    Currently, the schema of object in ArrayType or MapType is attached lazily, it will have better performance but introduce issues while serialization or accessing nested objects.
    
    This patch will apply schema to the objects of ArrayType or MapType immediately when accessing them, will be a little bit slower, but much robust.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2526 from davies/nested and squashes the following commits:
    
    2399ae5 [Davies Liu] fix serialization of List and Map in SchemaRDD