[SPARK-45980][INFRA] Add Python 3.11 to the Daily Python Github Action job

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to enable Python 3.11 testing in the following daily Python-only Github Action job.

https://github.com/apache/spark/actions/workflows/build_python.yml

Why are the changes needed?

To provide Python 3.11 test coverage to Apache Spark 4.0.0.

Since SPARK-45953 installed Python 3.11 into the infra image, what we need is to add it to the daily job.

• [SPARK-45979][INFRA] Add Python 3.11 to Infra docker image #43877

Does this PR introduce any user-facing change?

No.

How was this patch tested?

We need to validate this in the daily GitHub Action job.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @HyukjinKwon and @LuciferYang

[zhengruifeng]

@dongjoon-hyun many thanks for working on this!
BTW, do it make sense if we switch to anaconda (or miniconda) for better env management in the future? e.g. easily delete unrelated envs in a job to save disk space.

[dongjoon-hyun]

Thank you for review, @zhengruifeng .

That's a good point although I don't have a preference yet. Let's take a look at that later since it's just a distro transition while keeping the same test coverage.

[HyukjinKwon]

LGTM2!

[dongjoon-hyun]

Today's daily job shows Python 3.11 like the following.

========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['pypy3', 'python3.10', 'python3.11']

In addition, it exposes a python failure like the following. We will dig more and fix under the umbrella JIRA, SPARK-45981.

Had test failures in pyspark.pandas.tests.computation.test_apply_func with python3.11; see logs.