Skip to content
Snippets Groups Projects
Commit 3a22b100 authored by Holden Karau's avatar Holden Karau Committed by Cheng Lian
Browse files

[SPARK-10449] [SQL] Don't merge decimal types with incompatable precision or scales

From JIRA: Schema merging should only handle struct fields. But currently we also reconcile decimal precision and scale information.

Author: Holden Karau <holden@pigscanfly.ca>

Closes #8634 from holdenk/SPARK-10449-dont-merge-different-precision.
parent c6f8135e
No related branches found
No related tags found
No related merge requests found
...@@ -373,10 +373,19 @@ object StructType extends AbstractDataType { ...@@ -373,10 +373,19 @@ object StructType extends AbstractDataType {
StructType(newFields) StructType(newFields)
case (DecimalType.Fixed(leftPrecision, leftScale), case (DecimalType.Fixed(leftPrecision, leftScale),
DecimalType.Fixed(rightPrecision, rightScale)) => DecimalType.Fixed(rightPrecision, rightScale)) =>
DecimalType( if ((leftPrecision == rightPrecision) && (leftScale == rightScale)) {
max(leftScale, rightScale) + max(leftPrecision - leftScale, rightPrecision - rightScale), DecimalType(leftPrecision, leftScale)
max(leftScale, rightScale)) } else if ((leftPrecision != rightPrecision) && (leftScale != rightScale)) {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"precision $leftPrecision and $rightPrecision & scale $leftScale and $rightScale")
} else if (leftPrecision != rightPrecision) {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"precision $leftPrecision and $rightPrecision")
} else {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"scala $leftScale and $rightScale")
}
case (leftUdt: UserDefinedType[_], rightUdt: UserDefinedType[_]) case (leftUdt: UserDefinedType[_], rightUdt: UserDefinedType[_])
if leftUdt.userClass == rightUdt.userClass => leftUdt if leftUdt.userClass == rightUdt.userClass => leftUdt
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment