[SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas

[soxofaan]

What changes were proposed in this pull request?

See https://issues.apache.org/jira/browse/SPARK-41989 for in depth explanation

Short summary: pyspark/pandas/__init__.py uses, at import time, logging.warning() which might silently call logging.basicConfig().
So by importing pyspark.pandas (directly or indirectly) a user might unknowingly break their own logging setup (e.g. when based on logging.basicConfig() or related). logging.getLogger(...).warning() does not trigger this behavior.

Does this PR introduce any user-facing change?

User-defined logging setups will be more predictable.

How was this patch tested?

Manual testing so far.
I'm not sure it's worthwhile to cover this with a unit test

[HyukjinKwon]

Thanks for the fix @soxofaan. I think it was a mistake, cc @itholic

[HyukjinKwon]

I don't think there's any test for this, and the linter passed. Therefore, merging it now.

Merged to master.

[HyukjinKwon]

I merged it to branch-3.3 and branch-3.2 too.

[dongjoon-hyun]

Hi, @soxofaan and @HyukjinKwon .
This seems to cause a production down because some applications already rely on this import logging. After this patch, PySpark jobs fail.

[dongjoon-hyun]

FYI, cc @viirya , @huaxingao , @kazuyukitanimura , @sunchao

[HyukjinKwon]

Hmmm ... the change decouples logging from PySpark in order to do nothing with other logging systems, meaning that it removes the relation with the existing logging system. So I wonder how this patch affects things.

If it matters, we can try with the original fix proposed (https://github.com/apache/spark/compare/25aab030e562153dbf7d11e8d2dadd8fd10ecc50..242ff01536c6e8307224256faa5e3d197e780da9).

For a bit of context:

• PySpark haven't got its own logging system (for released versions). This was only place that logging was used in the released versions.
• In master branch, now for ML Torch integration and Spark Connect, we have them.

So, it would be helpful to know how this patch affected some PySpark jobs so we can properly fix them in master branch and other branches too.

[soxofaan]

@dongjoon-hyun

because some applications already rely on this import logging. After this patch, PySpark jobs fail.

Do you really mean that removing import logging from pyspark/pandas/__init__.py is the cause of breakage? So you have code that does from pyspark.pandas import logging instead of just import logging? Or maybe you have from pyspark.pandas import * and your implementation assumes this also imports logging?

Or do you mean that the removal of the logging.warn() call (and it's hidden logging.basicConfig side-effect) is causing breakage?

[dongjoon-hyun]

Thank you for the replies, @HyukjinKwon and @soxofaan . Please forget about my previous comments. After diggiing more about the PySpark 3.2 apps, it turns out that it was like a false alarm. Initially, there was a concern about the side-effect of the removed import statements, but this fix is a correct one and the affected applications should fix their python scripts if they are affected. Sorry about the false alarm.

[HyukjinKwon]

Thanks for confirming!