Skip to content
  • Xiangrui Meng's avatar
    1d4f3552
    [SPARK-3569][SQL] Add metadata field to StructField · 1d4f3552
    Xiangrui Meng authored
    Add `metadata: Metadata` to `StructField` to store extra information of columns. `Metadata` is a simple wrapper over `Map[String, Any]` with value types restricted to Boolean, Long, Double, String, Metadata, and arrays of those types. SerDe is via JSON.
    
    Metadata is preserved through simple operations like `SELECT`.
    
    marmbrus liancheng
    
    Author: Xiangrui Meng <meng@databricks.com>
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #2701 from mengxr/structfield-metadata and squashes the following commits:
    
    dedda56 [Xiangrui Meng] merge remote
    5ef930a [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    c35203f [Xiangrui Meng] Merge pull request #1 from marmbrus/pr/2701
    886b85c [Michael Armbrust] Expose Metadata and MetadataBuilder through the public scala and java packages.
    589f314 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    1e2abcf [Xiangrui Meng] change default value of metadata to None in python
    611d3c2 [Xiangrui Meng] move metadata from Expr to NamedExpr
    ddfcfad [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    a438440 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    4266f4d [Xiangrui Meng] add StructField.toString back for backward compatibility
    3f49aab [Xiangrui Meng] remove StructField.toString
    24a9f80 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    473a7c5 [Xiangrui Meng] merge master
    c9d7301 [Xiangrui Meng] organize imports
    1fcbf13 [Xiangrui Meng] change metadata type in StructField for Scala/Java
    60cc131 [Xiangrui Meng] add doc and header
    60614c7 [Xiangrui Meng] add metadata
    e42c452 [Xiangrui Meng] merge master
    93518fb [Xiangrui Meng] support metadata in python
    905bb89 [Xiangrui Meng] java conversions
    618e349 [Xiangrui Meng] make tests work in scala
    61b8e0f [Xiangrui Meng] merge master
    7e5a322 [Xiangrui Meng] do not output metadata in StructField.toString
    c41a664 [Xiangrui Meng] merge master
    d8af0ed [Xiangrui Meng] move tests to SQLQuerySuite
    67fdebb [Xiangrui Meng] add test on join
    d65072e [Xiangrui Meng] remove Map.empty
    367d237 [Xiangrui Meng] add test
    c194d5e [Xiangrui Meng] add metadata field to StructField and Attribute
    1d4f3552
    [SPARK-3569][SQL] Add metadata field to StructField
    Xiangrui Meng authored
    Add `metadata: Metadata` to `StructField` to store extra information of columns. `Metadata` is a simple wrapper over `Map[String, Any]` with value types restricted to Boolean, Long, Double, String, Metadata, and arrays of those types. SerDe is via JSON.
    
    Metadata is preserved through simple operations like `SELECT`.
    
    marmbrus liancheng
    
    Author: Xiangrui Meng <meng@databricks.com>
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #2701 from mengxr/structfield-metadata and squashes the following commits:
    
    dedda56 [Xiangrui Meng] merge remote
    5ef930a [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    c35203f [Xiangrui Meng] Merge pull request #1 from marmbrus/pr/2701
    886b85c [Michael Armbrust] Expose Metadata and MetadataBuilder through the public scala and java packages.
    589f314 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    1e2abcf [Xiangrui Meng] change default value of metadata to None in python
    611d3c2 [Xiangrui Meng] move metadata from Expr to NamedExpr
    ddfcfad [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    a438440 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    4266f4d [Xiangrui Meng] add StructField.toString back for backward compatibility
    3f49aab [Xiangrui Meng] remove StructField.toString
    24a9f80 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata
    473a7c5 [Xiangrui Meng] merge master
    c9d7301 [Xiangrui Meng] organize imports
    1fcbf13 [Xiangrui Meng] change metadata type in StructField for Scala/Java
    60cc131 [Xiangrui Meng] add doc and header
    60614c7 [Xiangrui Meng] add metadata
    e42c452 [Xiangrui Meng] merge master
    93518fb [Xiangrui Meng] support metadata in python
    905bb89 [Xiangrui Meng] java conversions
    618e349 [Xiangrui Meng] make tests work in scala
    61b8e0f [Xiangrui Meng] merge master
    7e5a322 [Xiangrui Meng] do not output metadata in StructField.toString
    c41a664 [Xiangrui Meng] merge master
    d8af0ed [Xiangrui Meng] move tests to SQLQuerySuite
    67fdebb [Xiangrui Meng] add test on join
    d65072e [Xiangrui Meng] remove Map.empty
    367d237 [Xiangrui Meng] add test
    c194d5e [Xiangrui Meng] add metadata field to StructField and Attribute
Loading