-
- Downloads
[SPARK-16157][SQL] Add New Methods for comments in StructField and StructType
#### What changes were proposed in this pull request? Based on the previous discussion with cloud-fan hvanhovell in another related PR https://github.com/apache/spark/pull/13764#discussion_r67994276, it looks reasonable to add convenience methods for users to add `comment` when defining `StructField`. Currently, the column-related `comment` attribute is stored in `Metadata` of `StructField`. For example, users can add the `comment` attribute using the following way: ```Scala StructType( StructField( "cl1", IntegerType, nullable = false, new MetadataBuilder().putString("comment", "test").build()) :: Nil) ``` This PR is to add more user friendly methods for the `comment` attribute when defining a `StructField`. After the changes, users are provided three different ways to do it: ```Scala val struct = (new StructType) .add("a", "int", true, "test1") val struct = (new StructType) .add("c", StringType, true, "test3") val struct = (new StructType) .add(StructField("d", StringType).withComment("test4")) ``` #### How was this patch tested? Added test cases: - `DataTypeSuite` is for testing three types of API changes, - `DataFrameReaderWriterSuite` is for parquet, json and csv formats - using in-memory catalog - `OrcQuerySuite.scala` is for orc format using Hive-metastore Author: gatorsmile <gatorsmile@gmail.com> Closes #13860 from gatorsmile/newMethodForComment.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 2 additions, 8 deletions...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructField.scala 18 additions, 0 deletions...c/main/scala/org/apache/spark/sql/types/StructField.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala 35 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/types/StructType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 17 additions, 0 deletions...test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 1 addition, 2 deletions...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandSuite.scala 1 addition, 2 deletions.../apache/spark/sql/execution/command/DDLCommandSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuite.scala 1 addition, 2 deletions...est/scala/org/apache/spark/sql/sources/DDLTestSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala 2 additions, 6 deletions...t/scala/org/apache/spark/sql/sources/TableScanSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala 26 additions, 1 deletion...rg/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala 22 additions, 0 deletions...t/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala
Loading
Please register or sign in to comment