Skip to content
Snippets Groups Projects
  • Ilya Ganelin's avatar
    f6fc254e
    [SPARK-8056][SQL] Design an easier way to construct schema for both Scala and Python · f6fc254e
    Ilya Ganelin authored
    I've added functionality to create new StructType similar to how we add parameters to a new SparkContext.
    
    I've also added tests for this type of creation.
    
    Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
    
    Closes #6686 from ilganeli/SPARK-8056B and squashes the following commits:
    
    27c1de1 [Ilya Ganelin] Rename
    467d836 [Ilya Ganelin] Removed from_string in favor of _parse_Datatype_json_value
    5fef5a4 [Ilya Ganelin] Updates for type parsing
    4085489 [Ilya Ganelin] Style errors
    3670cf5 [Ilya Ganelin] added string to DataType conversion
    8109e00 [Ilya Ganelin] Fixed error in tests
    41ab686 [Ilya Ganelin] Fixed style errors
    e7ba7e0 [Ilya Ganelin] Moved some python tests to tests.py. Added cleaner handling of null data type and added test for correctness of input format
    15868fa [Ilya Ganelin] Fixed python errors
    b79b992 [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-8056B
    a3369fc [Ilya Ganelin] Fixing space errors
    e240040 [Ilya Ganelin] Style
    bab7823 [Ilya Ganelin] Constructor error
    73d4677 [Ilya Ganelin] Style
    4ed00d9 [Ilya Ganelin] Fixed default arg
    67df57a [Ilya Ganelin] Removed Foo
    04cbf0c [Ilya Ganelin] Added comments for single object
    0484d7a [Ilya Ganelin] Restored second method
    6aeb740 [Ilya Ganelin] Style
    689e54d [Ilya Ganelin] Style
    f497e9e [Ilya Ganelin] Got rid of old code
    e3c7a88 [Ilya Ganelin] Fixed doctest failure
    a62ccde [Ilya Ganelin] Style
    966ac06 [Ilya Ganelin] style checks
    dabb7e6 [Ilya Ganelin] Added Python tests
    a3f4152 [Ilya Ganelin] added python bindings and better comments
    e6e536c [Ilya Ganelin] Added extra space
    7529a2e [Ilya Ganelin] Fixed formatting
    d388f86 [Ilya Ganelin] Fixed small bug
    c4e3bf5 [Ilya Ganelin] Reverted to using parse. Updated parse to support long
    d7634b6 [Ilya Ganelin] Reverted to fromString to properly support types
    22c39d5 [Ilya Ganelin] replaced FromString with DataTypeParser.parse. Replaced empty constructor initializing a null to have it instead create a new array to allow appends to it.
    faca398 [Ilya Ganelin] [SPARK-8056] Replaced default argument usage. Updated usage and code for DataType.fromString
    1acf76e [Ilya Ganelin] Scala style
    e31c674 [Ilya Ganelin] Fixed bug in test
    8dc0795 [Ilya Ganelin] Added tests for creation of StructType object with new methods
    fdf7e9f [Ilya Ganelin] [SPARK-8056] Created add methods to facilitate building new StructType objects.
    f6fc254e
    History
    [SPARK-8056][SQL] Design an easier way to construct schema for both Scala and Python
    Ilya Ganelin authored
    I've added functionality to create new StructType similar to how we add parameters to a new SparkContext.
    
    I've also added tests for this type of creation.
    
    Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
    
    Closes #6686 from ilganeli/SPARK-8056B and squashes the following commits:
    
    27c1de1 [Ilya Ganelin] Rename
    467d836 [Ilya Ganelin] Removed from_string in favor of _parse_Datatype_json_value
    5fef5a4 [Ilya Ganelin] Updates for type parsing
    4085489 [Ilya Ganelin] Style errors
    3670cf5 [Ilya Ganelin] added string to DataType conversion
    8109e00 [Ilya Ganelin] Fixed error in tests
    41ab686 [Ilya Ganelin] Fixed style errors
    e7ba7e0 [Ilya Ganelin] Moved some python tests to tests.py. Added cleaner handling of null data type and added test for correctness of input format
    15868fa [Ilya Ganelin] Fixed python errors
    b79b992 [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-8056B
    a3369fc [Ilya Ganelin] Fixing space errors
    e240040 [Ilya Ganelin] Style
    bab7823 [Ilya Ganelin] Constructor error
    73d4677 [Ilya Ganelin] Style
    4ed00d9 [Ilya Ganelin] Fixed default arg
    67df57a [Ilya Ganelin] Removed Foo
    04cbf0c [Ilya Ganelin] Added comments for single object
    0484d7a [Ilya Ganelin] Restored second method
    6aeb740 [Ilya Ganelin] Style
    689e54d [Ilya Ganelin] Style
    f497e9e [Ilya Ganelin] Got rid of old code
    e3c7a88 [Ilya Ganelin] Fixed doctest failure
    a62ccde [Ilya Ganelin] Style
    966ac06 [Ilya Ganelin] style checks
    dabb7e6 [Ilya Ganelin] Added Python tests
    a3f4152 [Ilya Ganelin] added python bindings and better comments
    e6e536c [Ilya Ganelin] Added extra space
    7529a2e [Ilya Ganelin] Fixed formatting
    d388f86 [Ilya Ganelin] Fixed small bug
    c4e3bf5 [Ilya Ganelin] Reverted to using parse. Updated parse to support long
    d7634b6 [Ilya Ganelin] Reverted to fromString to properly support types
    22c39d5 [Ilya Ganelin] replaced FromString with DataTypeParser.parse. Replaced empty constructor initializing a null to have it instead create a new array to allow appends to it.
    faca398 [Ilya Ganelin] [SPARK-8056] Replaced default argument usage. Updated usage and code for DataType.fromString
    1acf76e [Ilya Ganelin] Scala style
    e31c674 [Ilya Ganelin] Fixed bug in test
    8dc0795 [Ilya Ganelin] Added tests for creation of StructType object with new methods
    fdf7e9f [Ilya Ganelin] [SPARK-8056] Created add methods to facilitate building new StructType objects.