Skip to content

[SPARK-25461][PySpark][SQL] Add document for mismatch between return type of Pandas.Series and return type of pandas udf #22610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[SPARK-25461][PySpark][SQL] Add document for mismatch between return type of Pandas.Series and return type of pandas udf #22610

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2fa15bd
Ensure return type of Pandas.Series matches the arrow return type of …
viirya Sep 26, 2018
d206b7c
Print warning message instead of throwing exception.
viirya Oct 2, 2018
c084e74
Fix python style.
viirya Oct 2, 2018
a756c0b
Add document.
viirya Oct 4, 2018
6c6f8a1
Remove warning message and modify code comments.
viirya Oct 6, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions python/pyspark/sql/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2909,6 +2909,12 @@ def pandas_udf(f=None, returnType=None, functionType=None):
can fail on special rows, the workaround is to incorporate the condition into the functions.

.. note:: The user-defined functions do not take keyword arguments on the calling side.

.. note:: The data type of returned `pandas.Series` from the user-defined functions should be
matched with defined returnType (see :meth:`types.to_arrow_type` and
:meth:`types.from_arrow_type`). When there is mismatch between them, Spark might do
conversion on returned data. The conversion is not guaranteed to be correct and results
should be checked for accuracy by users.
Copy link
Member

@HyukjinKwon HyukjinKwon Oct 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am merging this since this describes the current status but let's make it clear and try to get rid of this note within 3.0.

Copy link
Member Author

@viirya viirya Oct 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agreed. If there is next upgrade of PyArrow available, we may be able to provide the option to raise error when an unsafe cast.

"""
# decorator @pandas_udf(returnType, functionType)
is_decorator = f is None or isinstance(f, (str, DataType))
Expand Down