Skip to content

Commit a81d792

Browse files
zhengruifengdongjoon-hyun
authored andcommitted
[SPARK-52680][DOCS][FOLLOW-UP] Update migration guide about Python 3.9 dropped
### What changes were proposed in this pull request? Update migration guide about Python 3.9 dropped ### Why are the changes needed? To guide end users ### Does this PR introduce _any_ user-facing change? yes, doc-only changes ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No Closes #51631 from zhengruifeng/doc_py_39. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 03cb4d9 commit a81d792

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

python/docs/source/migration_guide/pyspark_upgrade.rst

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,10 @@ Upgrading PySpark
2222
Upgrading from PySpark 4.0 to 4.1
2323
---------------------------------
2424

25+
* In Spark 4.1, Python 3.9 support was dropped in PySpark.
2526
* In Spark 4.1, ``DataFrame['name']`` on Spark Connect Python Client no longer eagerly validate the column name. To restore the legacy behavior, set ``PYSPARK_VALIDATE_COLUMN_NAME_LEGACY`` environment variable to ``1``.
26-
2727
* In Spark 4.1, Arrow-optimized Python UDF supports UDT input / output instead of falling back to the regular UDF. To restore the legacy behavior, set ``spark.sql.execution.pythonUDF.arrow.legacy.fallbackOnUDT`` to ``true``.
28-
2928
* In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled``.
30-
3129
* In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled``.
3230

3331
Upgrading from PySpark 3.5 to 4.0

0 commit comments

Comments
 (0)