-
- Downloads
[SPARK-9757] [SQL] Fixes persistence of Parquet relation with decimal column
PR #7967 enables us to save data source relations to metastore in Hive compatible format when possible. But it fails to persist Parquet relations with decimal column(s) to Hive metastore of versions lower than 1.2.0. This is because `ParquetHiveSerDe` in Hive versions prior to 1.2.0 doesn't support decimal. This PR checks for this case and falls back to Spark SQL specific metastore table format. Author: Yin Huai <yhuai@databricks.com> Author: Cheng Lian <lian@databricks.com> Closes #8130 from liancheng/spark-9757/old-hive-parquet-decimal.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayType.scala 5 additions, 1 deletion...src/main/scala/org/apache/spark/sql/types/ArrayType.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala 5 additions, 0 deletions.../src/main/scala/org/apache/spark/sql/types/DataType.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapType.scala 5 additions, 1 deletion...t/src/main/scala/org/apache/spark/sql/types/MapType.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala 6 additions, 2 deletions...rc/main/scala/org/apache/spark/sql/types/StructType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 24 additions, 0 deletions...test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 29 additions, 10 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientInterface.scala 3 additions, 0 deletions...la/org/apache/spark/sql/hive/client/ClientInterface.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/hive/client/ClientWrapper.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala 1 addition, 1 deletion...main/scala/org/apache/spark/sql/hive/client/package.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala 10 additions, 7 deletions...org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala 61 additions, 7 deletions...cala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
Loading
Please register or sign in to comment