Skip to content
Snippets Groups Projects
Commit 3a22b100 authored by Holden Karau's avatar Holden Karau Committed by Cheng Lian
Browse files

[SPARK-10449] [SQL] Don't merge decimal types with incompatable precision or scales

From JIRA: Schema merging should only handle struct fields. But currently we also reconcile decimal precision and scale information.

Author: Holden Karau <holden@pigscanfly.ca>

Closes #8634 from holdenk/SPARK-10449-dont-merge-different-precision.
parent c6f8135e
No related branches found
No related tags found
No related merge requests found
......@@ -373,10 +373,19 @@ object StructType extends AbstractDataType {
StructType(newFields)
case (DecimalType.Fixed(leftPrecision, leftScale),
DecimalType.Fixed(rightPrecision, rightScale)) =>
DecimalType(
max(leftScale, rightScale) + max(leftPrecision - leftScale, rightPrecision - rightScale),
max(leftScale, rightScale))
DecimalType.Fixed(rightPrecision, rightScale)) =>
if ((leftPrecision == rightPrecision) && (leftScale == rightScale)) {
DecimalType(leftPrecision, leftScale)
} else if ((leftPrecision != rightPrecision) && (leftScale != rightScale)) {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"precision $leftPrecision and $rightPrecision & scale $leftScale and $rightScale")
} else if (leftPrecision != rightPrecision) {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"precision $leftPrecision and $rightPrecision")
} else {
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
s"scala $leftScale and $rightScale")
}
case (leftUdt: UserDefinedType[_], rightUdt: UserDefinedType[_])
if leftUdt.userClass == rightUdt.userClass => leftUdt
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment