-
- Downloads
[SPARK-17183][SPARK-17983][SPARK-18101][SQL] put hive serde table schema to...
[SPARK-17183][SPARK-17983][SPARK-18101][SQL] put hive serde table schema to table properties like data source table ## What changes were proposed in this pull request? For data source tables, we will put its table schema, partition columns, etc. to table properties, to work around some hive metastore issues, e.g. not case-preserving, bad decimal type support, etc. We should also do this for hive serde tables, to reduce the difference between hive serde tables and data source tables, e.g. column names should be case preserving. ## How was this patch tested? existing tests, and a new test in `HiveExternalCatalog` Author: Wenchen Fan <wenchen@databricks.com> Closes #14750 from cloud-fan/minor1.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala 7 additions, 1 deletion...g/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala 0 additions, 6 deletions...g/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala 24 additions, 0 deletions.../src/main/scala/org/apache/spark/sql/types/DataType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala 20 additions, 0 deletions...che/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 5 additions, 5 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/execution/SparkSqlParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 4 additions, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala 3 additions, 1 deletion...in/scala/org/apache/spark/sql/execution/command/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 3 additions, 2 deletions...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 160 additions, 58 deletions...scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
- sql/hive/src/test/resources/golden/input1-2-d3aa54d5436b7b59ff5c7091b7ca6145 2 additions, 2 deletions...esources/golden/input1-2-d3aa54d5436b7b59ff5c7091b7ca6145
- sql/hive/src/test/resources/golden/input2-1-e0efeda558cd0194f4764a5735147b16 2 additions, 2 deletions...esources/golden/input2-1-e0efeda558cd0194f4764a5735147b16
- sql/hive/src/test/resources/golden/input2-2-aa9ab0598e0cb7a12c719f9b3d98dbfd 2 additions, 2 deletions...esources/golden/input2-2-aa9ab0598e0cb7a12c719f9b3d98dbfd
- sql/hive/src/test/resources/golden/input2-4-235f92683416fab031e6e7490487b15b 3 additions, 3 deletions...esources/golden/input2-4-235f92683416fab031e6e7490487b15b
- sql/hive/src/test/resources/golden/show_columns-2-b74990316ec4245fd8a7011e684b39da 3 additions, 3 deletions...es/golden/show_columns-2-b74990316ec4245fd8a7011e684b39da
- sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala 3 additions, 6 deletions...pache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 2 additions, 2 deletions...a/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
Loading
Please register or sign in to comment