Skip to content

Commit 4051fff

Browse files
holdenkliancheng
authored andcommitted
[SPARK-10449] [SQL] Don't merge decimal types with incompatable precision or scales
From JIRA: Schema merging should only handle struct fields. But currently we also reconcile decimal precision and scale information. Author: Holden Karau <[email protected]> Closes #8634 from holdenk/SPARK-10449-dont-merge-different-precision. (cherry picked from commit 3a22b10) Signed-off-by: Cheng Lian <[email protected]>
1 parent 3df52cc commit 4051fff

File tree

1 file changed

+13
-4
lines changed

1 file changed

+13
-4
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -373,10 +373,19 @@ object StructType extends AbstractDataType {
373373
StructType(newFields)
374374

375375
case (DecimalType.Fixed(leftPrecision, leftScale),
376-
DecimalType.Fixed(rightPrecision, rightScale)) =>
377-
DecimalType(
378-
max(leftScale, rightScale) + max(leftPrecision - leftScale, rightPrecision - rightScale),
379-
max(leftScale, rightScale))
376+
DecimalType.Fixed(rightPrecision, rightScale)) =>
377+
if ((leftPrecision == rightPrecision) && (leftScale == rightScale)) {
378+
DecimalType(leftPrecision, leftScale)
379+
} else if ((leftPrecision != rightPrecision) && (leftScale != rightScale)) {
380+
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
381+
s"precision $leftPrecision and $rightPrecision & scale $leftScale and $rightScale")
382+
} else if (leftPrecision != rightPrecision) {
383+
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
384+
s"precision $leftPrecision and $rightPrecision")
385+
} else {
386+
throw new SparkException("Failed to merge Decimal Tpes with incompatible " +
387+
s"scala $leftScale and $rightScale")
388+
}
380389

381390
case (leftUdt: UserDefinedType[_], rightUdt: UserDefinedType[_])
382391
if leftUdt.userClass == rightUdt.userClass => leftUdt

0 commit comments

Comments
 (0)