Skip to content

Commit aa3a742

Browse files
MaxGekkHyukjinKwon
authored andcommitted
[SPARK-31159][SQL][FOLLOWUP] Move checking of the rebaseDateTime flag out of the loop in VectorizedColumnReader
### What changes were proposed in this pull request? In the PR, I propose to refactor reading of timestamps of the `TIMESTAMP_MILLIS` logical type from Parquet files in `VectorizedColumnReader`, and move checking of the `rebaseDateTime` flag out of the internal loop. ### Why are the changes needed? To avoid any additional overhead of the checking the SQL config `spark.sql.legacy.parquet.rebaseDateTime.enabled` introduced by the PR #27915. ### Does this PR introduce any user-facing change? No ### How was this patch tested? By running the test suite `ParquetIOSuite`. Closes #27973 from MaxGekk/rebase-parquet-datetime-followup. Authored-by: Maxim Gekk <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent d929c0d commit aa3a742

File tree

1 file changed

+15
-8
lines changed

1 file changed

+15
-8
lines changed

sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -462,15 +462,22 @@ private void readLongBatch(int rowId, int num, WritableColumnVector column) thro
462462
num, column, rowId, maxDefLevel, (VectorizedValuesReader) dataColumn);
463463
}
464464
} else if (originalType == OriginalType.TIMESTAMP_MILLIS) {
465-
for (int i = 0; i < num; i++) {
466-
if (defColumn.readInteger() == maxDefLevel) {
467-
long micros = DateTimeUtils.millisToMicros(dataColumn.readLong());
468-
if (rebaseDateTime) {
469-
micros = DateTimeUtils.rebaseJulianToGregorianMicros(micros);
465+
if (rebaseDateTime) {
466+
for (int i = 0; i < num; i++) {
467+
if (defColumn.readInteger() == maxDefLevel) {
468+
long micros = DateTimeUtils.millisToMicros(dataColumn.readLong());
469+
column.putLong(rowId + i, DateTimeUtils.rebaseJulianToGregorianMicros(micros));
470+
} else {
471+
column.putNull(rowId + i);
472+
}
473+
}
474+
} else {
475+
for (int i = 0; i < num; i++) {
476+
if (defColumn.readInteger() == maxDefLevel) {
477+
column.putLong(rowId + i, DateTimeUtils.millisToMicros(dataColumn.readLong()));
478+
} else {
479+
column.putNull(rowId + i);
470480
}
471-
column.putLong(rowId + i, micros);
472-
} else {
473-
column.putNull(rowId + i);
474481
}
475482
}
476483
} else {

0 commit comments

Comments
 (0)