-
- Downloads
[SPARK-11856][SQL] add type cast if the real type is different but compatible with encoder schema
When we build the `fromRowExpression` for an encoder, we set up a lot of "unresolved" stuff and lost the required data type, which may lead to runtime error if the real type doesn't match the encoder's schema. For example, we build an encoder for `case class Data(a: Int, b: String)` and the real type is `[a: int, b: long]`, then we will hit runtime error and say that we can't construct class `Data` with int and long, because we lost the information that `b` should be a string. Author: Wenchen Fan <wenchen@databricks.com> Closes #9840 from cloud-fan/err-msg.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 74 additions, 19 deletions...scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 40 additions, 0 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala 1 addition, 1 deletion...apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala 3 additions, 1 deletion...pache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala 9 additions, 0 deletions...cala/org/apache/spark/sql/catalyst/expressions/Cast.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala 1 addition, 1 deletion...e/spark/sql/catalyst/expressions/complexTypeCreator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala 12 additions, 0 deletions...c/main/scala/org/apache/spark/sql/types/DecimalType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala 180 additions, 0 deletions.../spark/sql/catalyst/encoders/EncoderResolutionSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala 2 additions, 2 deletions...t/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 13 additions, 8 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment