-
- Downloads
[SPARK-15657][SQL] RowEncoder should validate the data type of input object
## What changes were proposed in this pull request? This PR improves the error handling of `RowEncoder`. When we create a `RowEncoder` with a given schema, we should validate the data type of input object. e.g. we should throw an exception when a field is boolean but is declared as a string column. This PR also removes the support to use `Product` as a valid external type of struct type. This support is added at https://github.com/apache/spark/pull/9712, but is incomplete, e.g. nested product, product in array are both not working. However, we never officially support this feature and I think it's ok to ban it. ## How was this patch tested? new tests in `RowEncoderSuite`. Author: Wenchen Fan <wenchen@databricks.com> Closes #13401 from cloud-fan/bug.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala 1 addition, 9 deletionssql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala 11 additions, 6 deletions...a/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala 52 additions, 9 deletions...ache/spark/sql/catalyst/expressions/objects/objects.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala 31 additions, 16 deletions.../apache/spark/sql/catalyst/encoders/RowEncoderSuite.scala
Loading
Please register or sign in to comment