Skip to content

Conversation

@zhengruifeng
Copy link
Contributor

@zhengruifeng zhengruifeng commented Sep 18, 2024

What changes were proposed in this pull request?

Refine the string representation of literal date and datetime

Why are the changes needed?

1, we should not represent those literals with internal values;
2, the string representation should be consistent with PySpark Classic if possible (we cannot make sure the representations are always the same because we only hold an unresolved expression in connect, but we can try our best to do so)

Does this PR introduce any user-facing change?

yes

before:

In [3]: lit(datetime.date(2024, 7, 10))
Out[3]: Column<'19914'>

In [4]: lit(datetime.datetime(2024, 7, 10, 1, 2, 3, 456))
Out[4]: Column<'1720544523000456'>

after:

In [3]: lit(datetime.date(2024, 7, 10))
Out[3]: Column<'2024-07-10'>

In [4]: lit(datetime.datetime(2024, 7, 10, 1, 2, 3, 456))
Out[4]: Column<'2024-07-10 01:02:03.000456'>

How was this patch tested?

added tests

Was this patch authored or co-authored using generative AI tooling?

no

ts = TimestampNTZType().fromInternal(self._value)
if ts is not None and isinstance(ts, datetime.datetime):
return ts.strftime("%Y-%m-%d %H:%M:%S.%f")
# TODO(SPARK-49693): Refine the string representation of timedelta
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't find a standard method to convert timedelta to a ISO string like the JVM side

@zhengruifeng zhengruifeng changed the title [SPARK-49692][PYTHON][CONNECT] Refine the string representation of literal date and timestamp [SPARK-49692][PYTHON][CONNECT] Refine the string representation of literal date and datetime Sep 18, 2024
@zhengruifeng
Copy link
Contributor Author

thanks @xinrong-meng , merged to master

@zhengruifeng zhengruifeng deleted the py_connect_lit_dt branch September 18, 2024 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants