-
- Downloads
[SPARK-17335][SQL] Fix ArrayType and MapType CatalogString.
## What changes were proposed in this pull request? the `catalogString` for `ArrayType` and `MapType` currently calls the `simpleString` method on its children. This is a problem when the child is a struct, the `struct.simpleString` implementation truncates the number of fields it shows (25 at max). This breaks the generation of a proper `catalogString`, and has shown to cause errors while writing to Hive. This PR fixes this by providing proper `catalogString` implementations for `ArrayData` or `MapData`. ## How was this patch tested? Added testing for `catalogString` to `DataTypeSuite`. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #14938 from hvanhovell/SPARK-17335.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayType.scala 2 additions, 0 deletions...src/main/scala/org/apache/spark/sql/types/ArrayType.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapType.scala 2 additions, 0 deletions...t/src/main/scala/org/apache/spark/sql/types/MapType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 30 additions, 0 deletions...test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
- sql/core/benchmarks/WideSchemaBenchmark-results.txt 99 additions, 75 deletionssql/core/benchmarks/WideSchemaBenchmark-results.txt
Loading
Please register or sign in to comment