Skip to content

Commit 94e2fce

Browse files
committed
address comments
1 parent 9caf98c commit 94e2fce

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

docs/sql-migration-guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ license: |
113113

114114
### UDFs and Built-in Functions
115115

116-
- Since Spark 3.0, the `date_add` and `date_sub` functions only accepts int, smallint, tinyint as the 2nd argument, fractional and string types are not valid anymore, e.g. `date_add(cast('1964-05-23' as date), '12.34')` will cause `AnalysisException`. Note that, string literals are still allowed, but Spark will throw Analysis Exception if the string is not a valid integer. In Spark version 2.4 and earlier, if the 2nd argument is fractional or string value, it will be coerced to int value, and the result will be a date value of `1964-06-04`.
116+
- Since Spark 3.0, the `date_add` and `date_sub` functions only accept int, smallint, tinyint as the 2nd argument, fractional and non-literal string are not valid anymore, e.g. `date_add(cast('1964-05-23' as date), 12.34)` will cause `AnalysisException`. Note that, string literals are still allowed, but Spark will throw Analysis Exception if the string content is not a valid integer. In Spark version 2.4 and earlier, if the 2nd argument is fractional or string value, it will be coerced to int value, and the result will be a date value of `1964-06-04`.
117117

118118
- Since Spark 3.0, the function `percentile_approx` and its alias `approx_percentile` only accept integral value with range in `[1, 2147483647]` as its 3rd argument `accuracy`, fractional and string types are disallowed, e.g. `percentile_approx(10.0, 0.2, 1.8D)` will cause `AnalysisException`. In Spark version 2.4 and earlier, if `accuracy` is fractional or string value, it will be coerced to an int value, `percentile_approx(10.0, 0.2, 1.8D)` is operated as `percentile_approx(10.0, 0.2, 1)` which results in `10.0`.
119119

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1049,7 +1049,7 @@ object TypeCoercion {
10491049
/**
10501050
* A special rule to support string literal as the second argument of date_add/date_sub functions,
10511051
* to keep backward compatibility as a temporary workaround.
1052-
* TODO: revisit the type coercion rules for string.
1052+
* TODO(SPARK-28589): implement ANSI type type coercion and handle string literals.
10531053
*/
10541054
object StringLiteralCoercion extends TypeCoercionRule {
10551055
override protected def coerceTypes(plan: LogicalPlan): LogicalPlan = plan resolveExpressions {

0 commit comments

Comments
 (0)