Skip to content
  • Herman van Hovell's avatar
    c2a1576c
    [SPARK-17335][SQL] Fix ArrayType and MapType CatalogString. · c2a1576c
    Herman van Hovell authored
    ## What changes were proposed in this pull request?
    the `catalogString` for `ArrayType` and `MapType` currently calls the `simpleString` method on its children. This is a problem when the child is a struct, the `struct.simpleString` implementation truncates the number of fields it shows (25 at max). This breaks the generation of a proper `catalogString`, and has shown to cause errors while writing to Hive.
    
    This PR fixes this by providing proper `catalogString` implementations for `ArrayData` or `MapData`.
    
    ## How was this patch tested?
    Added testing for `catalogString` to `DataTypeSuite`.
    
    Author: Herman van Hovell <hvanhovell@databricks.com>
    
    Closes #14938 from hvanhovell/SPARK-17335.
    c2a1576c
    [SPARK-17335][SQL] Fix ArrayType and MapType CatalogString.
    Herman van Hovell authored
    ## What changes were proposed in this pull request?
    the `catalogString` for `ArrayType` and `MapType` currently calls the `simpleString` method on its children. This is a problem when the child is a struct, the `struct.simpleString` implementation truncates the number of fields it shows (25 at max). This breaks the generation of a proper `catalogString`, and has shown to cause errors while writing to Hive.
    
    This PR fixes this by providing proper `catalogString` implementations for `ArrayData` or `MapData`.
    
    ## How was this patch tested?
    Added testing for `catalogString` to `DataTypeSuite`.
    
    Author: Herman van Hovell <hvanhovell@databricks.com>
    
    Closes #14938 from hvanhovell/SPARK-17335.
Loading