-
- Downloads
[SPARK-3569][SQL] Add metadata field to StructField
Add `metadata: Metadata` to `StructField` to store extra information of columns. `Metadata` is a simple wrapper over `Map[String, Any]` with value types restricted to Boolean, Long, Double, String, Metadata, and arrays of those types. SerDe is via JSON. Metadata is preserved through simple operations like `SELECT`. marmbrus liancheng Author: Xiangrui Meng <meng@databricks.com> Author: Michael Armbrust <michael@databricks.com> Closes #2701 from mengxr/structfield-metadata and squashes the following commits: dedda56 [Xiangrui Meng] merge remote 5ef930a [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata c35203f [Xiangrui Meng] Merge pull request #1 from marmbrus/pr/2701 886b85c [Michael Armbrust] Expose Metadata and MetadataBuilder through the public scala and java packages. 589f314 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata 1e2abcf [Xiangrui Meng] change default value of metadata to None in python 611d3c2 [Xiangrui Meng] move metadata from Expr to NamedExpr ddfcfad [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata a438440 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata 4266f4d [Xiangrui Meng] add StructField.toString back for backward compatibility 3f49aab [Xiangrui Meng] remove StructField.toString 24a9f80 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into structfield-metadata 473a7c5 [Xiangrui Meng] merge master c9d7301 [Xiangrui Meng] organize imports 1fcbf13 [Xiangrui Meng] change metadata type in StructField for Scala/Java 60cc131 [Xiangrui Meng] add doc and header 60614c7 [Xiangrui Meng] add metadata e42c452 [Xiangrui Meng] merge master 93518fb [Xiangrui Meng] support metadata in python 905bb89 [Xiangrui Meng] java conversions 618e349 [Xiangrui Meng] make tests work in scala 61b8e0f [Xiangrui Meng] merge master 7e5a322 [Xiangrui Meng] do not output metadata in StructField.toString c41a664 [Xiangrui Meng] merge master d8af0ed [Xiangrui Meng] move tests to SQLQuerySuite 67fdebb [Xiangrui Meng] add test on join d65072e [Xiangrui Meng] remove Map.empty 367d237 [Xiangrui Meng] add test c194d5e [Xiangrui Meng] add metadata field to StructField and Attribute
Showing
- python/pyspark/sql.py 11 additions, 4 deletionspython/pyspark/sql.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala 1 addition, 0 deletions...rg/apache/spark/sql/catalyst/expressions/Expression.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala 1 addition, 1 deletion...rg/apache/spark/sql/catalyst/expressions/generators.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala 24 additions, 7 deletions...che/spark/sql/catalyst/expressions/namedExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala 18 additions, 7 deletions...scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/Metadata.scala 255 additions, 0 deletions...n/scala/org/apache/spark/sql/catalyst/util/Metadata.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/MetadataSuite.scala 82 additions, 0 deletions...la/org/apache/spark/sql/catalyst/util/MetadataSuite.scala
- sql/core/src/main/java/org/apache/spark/sql/api/java/DataType.java 19 additions, 6 deletions...src/main/java/org/apache/spark/sql/api/java/DataType.java
- sql/core/src/main/java/org/apache/spark/sql/api/java/Metadata.java 31 additions, 0 deletions...src/main/java/org/apache/spark/sql/api/java/Metadata.java
- sql/core/src/main/java/org/apache/spark/sql/api/java/MetadataBuilder.java 28 additions, 0 deletions...n/java/org/apache/spark/sql/api/java/MetadataBuilder.java
- sql/core/src/main/java/org/apache/spark/sql/api/java/StructField.java 17 additions, 2 deletions.../main/java/org/apache/spark/sql/api/java/StructField.java
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 0 additions, 1 deletion...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala 4 additions, 9 deletions...re/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/package.scala 23 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/package.scala
- sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala 5 additions, 3 deletions...org/apache/spark/sql/types/util/DataTypeConversions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataTypeSuite.scala 5 additions, 3 deletions...e/src/test/scala/org/apache/spark/sql/DataTypeSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 30 additions, 7 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/TestData.scala 11 additions, 0 deletionssql/core/src/test/scala/org/apache/spark/sql/TestData.scala
- sql/core/src/test/scala/org/apache/spark/sql/api/java/ScalaSideDataTypeConversionSuite.scala 7 additions, 5 deletions...spark/sql/api/java/ScalaSideDataTypeConversionSuite.scala
Loading
Please register or sign in to comment