-
- Downloads
[SPARK-19459][SQL] Add Hive datatype (char/varchar) to StructField metadata
## What changes were proposed in this pull request? Reading from an existing ORC table which contains `char` or `varchar` columns can fail with a `ClassCastException` if the table metadata has been created using Spark. This is caused by the fact that spark internally replaces `char` and `varchar` columns with a `string` column. This PR fixes this by adding the hive type to the `StructField's` metadata under the `HIVE_TYPE_STRING` key. This is picked up by the `HiveClient` and the ORC reader, see https://github.com/apache/spark/pull/16060 for more details on how the metadata is used. ## How was this patch tested? Added a regression test to `OrcSourceSuite`. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #16804 from hvanhovell/SPARK-19459.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 22 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/package.scala 9 additions, 1 deletion...t/src/main/scala/org/apache/spark/sql/types/package.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala 6 additions, 2 deletions...t/scala/org/apache/spark/sql/sources/TableScanSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala 3 additions, 11 deletions.../src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 4 additions, 4 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala 32 additions, 5 deletions.../scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala
Loading
Please register or sign in to comment