Skip to content
  • Cheng Lian's avatar
    a42cc08d
    [SPARK-3713][SQL] Uses JSON to serialize DataType objects · a42cc08d
    Cheng Lian authored
    This PR uses JSON instead of `toString` to serialize `DataType`s. The latter is not only hard to parse but also flaky in many cases.
    
    Since we already write schema information to Parquet metadata in the old style, we have to reserve the old `DataType` parser and ensure downward compatibility. The old parser is now renamed to `CaseClassStringParser` and moved into `object DataType`.
    
    JoshRosen davies Please help review PySpark related changes, thanks!
    
    Author: Cheng Lian <lian.cs.zju@gmail.com>
    
    Closes #2563 from liancheng/datatype-to-json and squashes the following commits:
    
    fc92eb3 [Cheng Lian] Reverts debugging code, simplifies primitive type JSON representation
    438c75f [Cheng Lian] Refactors PySpark DataType JSON SerDe per comments
    6b6387b [Cheng Lian] Removes debugging code
    6a3ee3a [Cheng Lian] Addresses per review comments
    dc158b5 [Cheng Lian] Addresses PEP8 issues
    99ab4ee [Cheng Lian] Adds compatibility est case for Parquet type conversion
    a983a6c [Cheng Lian] Adds PySpark support
    f608c6e [Cheng Lian] De/serializes DataType objects from/to JSON
    a42cc08d
    [SPARK-3713][SQL] Uses JSON to serialize DataType objects
    Cheng Lian authored
    This PR uses JSON instead of `toString` to serialize `DataType`s. The latter is not only hard to parse but also flaky in many cases.
    
    Since we already write schema information to Parquet metadata in the old style, we have to reserve the old `DataType` parser and ensure downward compatibility. The old parser is now renamed to `CaseClassStringParser` and moved into `object DataType`.
    
    JoshRosen davies Please help review PySpark related changes, thanks!
    
    Author: Cheng Lian <lian.cs.zju@gmail.com>
    
    Closes #2563 from liancheng/datatype-to-json and squashes the following commits:
    
    fc92eb3 [Cheng Lian] Reverts debugging code, simplifies primitive type JSON representation
    438c75f [Cheng Lian] Refactors PySpark DataType JSON SerDe per comments
    6b6387b [Cheng Lian] Removes debugging code
    6a3ee3a [Cheng Lian] Addresses per review comments
    dc158b5 [Cheng Lian] Addresses PEP8 issues
    99ab4ee [Cheng Lian] Adds compatibility est case for Parquet type conversion
    a983a6c [Cheng Lian] Adds PySpark support
    f608c6e [Cheng Lian] De/serializes DataType objects from/to JSON
Loading