-
- Downloads
[SPARK-18220][SQL] read Hive orc table with varchar column should not fail
## What changes were proposed in this pull request? Spark SQL only has `StringType`, when reading hive table with varchar column, we will read that column as `StringType`. However, we still need to use varchar `ObjectInspector` to read varchar column in hive table, which means we need to know the actual column type at hive side. In Spark 2.1, after https://github.com/apache/spark/pull/14363 , we parse hive type string to catalyst type, which means the actual column type at hive side is erased. Then we may use string `ObjectInspector` to read varchar column and fail. This PR keeps the original hive column type string in the metadata of `StructField`, and use it when we convert it to a hive column. ## How was this patch tested? newly added regression test Author: Wenchen Fan <wenchen@databricks.com> Closes #16060 from cloud-fan/varchar.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala 8 additions, 0 deletions.../src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/MetastoreRelation.scala 6 additions, 1 deletion...n/scala/org/apache/spark/sql/hive/MetastoreRelation.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 12 additions, 3 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogBackwardCompatibilitySuite.scala 2 additions, 2 deletions.../hive/HiveExternalCatalogBackwardCompatibilitySuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala 12 additions, 0 deletions.../scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala
Loading
Please register or sign in to comment