[SPARK-49693][PYTHON][CONNECT] Refine the string representation of timedelta

[zhengruifeng]

What changes were proposed in this pull request?

Refine the string representation of timedelta, by following the ISO format.
Note that the used units in JVM side (Duration) and Pandas are different.

Why are the changes needed?

We should not leak the raw data

Does this PR introduce any user-facing change?

yes

PySpark Classic:

In [1]: from pyspark.sql import functions as sf

In [2]: import datetime

In [3]: sf.lit(datetime.timedelta(1, 1))
Out[3]: Column<'PT24H1S'>

PySpark Connect (before):

In [1]: from pyspark.sql import functions as sf

In [2]: import datetime

In [3]: sf.lit(datetime.timedelta(1, 1))
Out[3]: Column<'86401000000'>

PySpark Connect (after):

In [1]: from pyspark.sql import functions as sf

In [2]: import datetime

In [3]: sf.lit(datetime.timedelta(1, 1))
Out[3]: Column<'P1DT0H0M1S'>

How was this patch tested?

added test

Was this patch authored or co-authored using generative AI tooling?

no

[HyukjinKwon]

spark connect requires pyarrow/pandas iirc so you won't need to handle exceptions here if that's what you're covering

[zhengruifeng]

yeah, the pandas is a mandatory dependency for connect, let me remove this try-catch

[HyukjinKwon]

I think it should be connect specific test?

[zhengruifeng]

it also works for pyspark classic.

Classic also use a ISO-8601 string, but JVM side and Pandas apply different units.

A string representation from the JVM side can also be parsed by Pandas.

[zhengruifeng]

This test will be ran in both classic and connect

[zhengruifeng]

thanks, merged to master