[SPARK-49082][SQL] Support widening Date to TimestampNTZ in Avro reader

[aldenlau-db]

What changes were proposed in this pull request?

This change adds support for widening type promotions from Date to TimestampNTZ in `AvroDeserializer. This PR is a follow-up to #47582 which adds support for other widening type promotions.

Why are the changes needed?

When reading Avro files with a mix of Date and TimestampNTZ for a given column, the reader should be able to read all files and promote Date to TimestampNTZ instead of throwing an error when reading files with Date.

Although SPARK-49082 was resolved by #47582, that PR did not include Date -> TimestampNTZ widening. The change in this PR is very similar to #44368 which adds support for Date -> TimestampNTZ widening for the Parquet reader.

Does this PR introduce any user-facing change?

Yes, users will no longer see an error when attempting to read a file containing Date when the read schema contains TimestampNTZ. The time will be set to 00:00, as has been done in #44368.

How was this patch tested?

New test in AvroSuite.

Was this patch authored or co-authored using generative AI tooling?

No

[aldenlau-db]

cc @johanl-db @sandip-db

[aldenlau-db]

@johanl-db These test cases fail with an ArithmeticException. However, this implementation is based on the Parquet implementation. I noticed that the Parquet reader will also fail when attempting to upcast any date earlier than -290308-12-22 BCE and later than +294247-01-10 CE with the same ArithmeticException due to long overflow.

I think this is because the Date Spark type supports dates from June 23 -5877641 CE to July 11 +5881580 CE, but TimestampNTZ supports -290308-12-21 BCE 19:59:06 to +294247-01-10 CE 04:00:54, which is a smaller range. In the context of type widening, shouldn't this widening be unsupported by all readers since Date stores a larger range of values than TimestampNTZ (even though it has less precision)?

[johanl-db]

That's not great..

I think for now we can document that limitation. Iceberg allows widening date -> timestamp without timezone and also calls this out: https://iceberg.apache.org/spec/#schema-evolution
Hopefully very few users are using date values outside -290308-12-21 BCE - +294247-01-10 CE

Moving forward, we'll probably want to address this by moving the error from read time to write time, i.e. check that all values in the table fit the timestamp range before applying the type change (which we should be able to tell from stats only in most cases).

Note: this is a Delta issue really, so not issue for this spark PR in itself

[aldenlau-db]

Makes sense, thanks. I have updated the test cases in this PR and added a comment documenting this behavior.

[HyukjinKwon]

Merged to master.