Commit c9fcfd5
[SPARK-25461][PYSPARK][SQL] Add document for mismatch between return type of Pandas.Series and return type of pandas udf
## What changes were proposed in this pull request?
For Pandas UDFs, we get arrow type from defined Catalyst return data type of UDFs. We use this arrow type to do serialization of data. If the defined return data type doesn't match with actual return type of Pandas.Series returned by Pandas UDFs, it has a risk to return incorrect data from Python side.
Currently we don't have reliable approach to check if the data conversion is safe or not. We leave some document to notify this to users for now. When there is next upgrade of PyArrow available we can use to check it, we should add the option to check it.
## How was this patch tested?
Only document change.
Closes apache#22610 from viirya/SPARK-25461.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: hyukjinkwon <[email protected]>1 parent 35e2c01 commit c9fcfd5
1 file changed
+6
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2948 | 2948 | | |
2949 | 2949 | | |
2950 | 2950 | | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
2951 | 2957 | | |
2952 | 2958 | | |
2953 | 2959 | | |
| |||
0 commit comments