Skip to content

Commit 2111b59

Browse files
committed
[SPARK-27624][CORE] Fix CalenderInterval to show an empty interval correctly
## What changes were proposed in this pull request? If the interval is `0`, it doesn't show both the value `0` and the unit at all. For example, this happens in the explain plans and Spark Web UI on `EventTimeWatermark` diagram. **BEFORE** ```scala scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain == Physical Plan == EventTimeWatermark ts#0: timestamp, interval 1 microseconds +- StreamingRelation FileSource[/tmp/t], [ts#0] scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain == Physical Plan == EventTimeWatermark ts#3: timestamp, interval +- StreamingRelation FileSource[/tmp/t], [ts#3] ``` **AFTER** ```scala scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "1 microsecond").explain == Physical Plan == EventTimeWatermark ts#0: timestamp, interval 1 microseconds +- StreamingRelation FileSource[/tmp/t], [ts#0] scala> spark.readStream.schema("ts timestamp").parquet("/tmp/t").withWatermark("ts", "0 microsecond").explain == Physical Plan == EventTimeWatermark ts#3: timestamp, interval 0 microseconds +- StreamingRelation FileSource[/tmp/t], [ts#3] ``` ## How was this patch tested? Pass the Jenkins with the updated test case. Closes #24516 from dongjoon-hyun/SPARK-27624. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 614a5cc) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent b3d30a8 commit 2111b59

File tree

2 files changed

+5
-0
lines changed

2 files changed

+5
-0
lines changed

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,8 @@ public String toString() {
319319
appendUnit(sb, rest / MICROS_PER_MILLI, "millisecond");
320320
rest %= MICROS_PER_MILLI;
321321
appendUnit(sb, rest, "microsecond");
322+
} else if (months == 0) {
323+
sb.append(" 0 microseconds");
322324
}
323325

324326
return sb.toString();

common/unsafe/src/test/java/org/apache/spark/unsafe/types/CalendarIntervalSuite.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,9 @@ public void equalsTest() {
4141
public void toStringTest() {
4242
CalendarInterval i;
4343

44+
i = new CalendarInterval(0, 0);
45+
assertEquals("interval 0 microseconds", i.toString());
46+
4447
i = new CalendarInterval(34, 0);
4548
assertEquals("interval 2 years 10 months", i.toString());
4649

0 commit comments

Comments
 (0)