You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas
See https://issues.apache.org/jira/browse/SPARK-41989 for in depth explanation
Short summary: `pyspark/pandas/__init__.py` uses, at import time, `logging.warning()` which might silently call `logging.basicConfig()`.
So by importing `pyspark.pandas` (directly or indirectly) a user might unknowingly break their own logging setup (e.g. when based on `logging.basicConfig()` or related). `logging.getLogger(...).warning()` does not trigger this behavior.
User-defined logging setups will be more predictable.
Manual testing so far.
I'm not sure it's worthwhile to cover this with a unit test
Closesapache#39516 from soxofaan/SPARK-41989-pyspark-pandas-logging-setup.
Authored-by: Stefaan Lippens <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 04836ba)
Signed-off-by: Hyukjin Kwon <[email protected]>
0 commit comments