Commit a27f021
[SPARK-36076][SQL][3.1] ArrayIndexOutOfBounds in Cast string to times…
…tamp
### What changes were proposed in this pull request?
Casting string to timestamp might throw ArrayIndexOutOfBounds in certain cases
This error only occur in branch 3.0, 3.1 and previous, it's not present on 3.2 or master
Code to reproduce:
```
val df = Seq(":8:434421+ 98:38").toDF("c0")
val df2 = df.withColumn("c1", col("c0").cast(DataTypes.TimestampType))
df2.show()
```
Error:
```
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.stringToTimestamp(DateTimeUtils.scala:328)
at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToTimestamp$2(Cast.scala:455)
at org.apache.spark.sql.catalyst.expressions.CastBase.buildCast(Cast.scala:295)
at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToTimestamp$1(Cast.scala:451)
at org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:840)
at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:476)
```
### Why are the changes needed?
Cast String to timestamp shouldn't throw error, it should return Null instead.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add test in DateTimeUtilsSuite
Closes apache#33293 from dgd-contributor/SPARK-36076_CastStringToTimeStampThrowArrayIndexOutOfBoundsException.
Authored-by: dgd-contributor <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>1 parent ccfbc17 commit a27f021
File tree
2 files changed
+23
-21
lines changed- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/util
- test/scala/org/apache/spark/sql/catalyst/util
2 files changed
+23
-21
lines changedLines changed: 19 additions & 21 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
242 | 242 | | |
243 | 243 | | |
244 | 244 | | |
245 | | - | |
| 245 | + | |
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
| |||
350 | 350 | | |
351 | 351 | | |
352 | 352 | | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
373 | 365 | | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
374 | 372 | | |
375 | 373 | | |
376 | 374 | | |
| |||
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
283 | 283 | | |
284 | 284 | | |
285 | 285 | | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
286 | 290 | | |
287 | 291 | | |
288 | 292 | | |
| |||
0 commit comments