[SPARK-45639][SQL][PYTHON] Support loading Python data sources in DataFrameReader

[allisonwang-db]

What changes were proposed in this pull request?

This PR supports spark.read.format(...).load() for Python data sources.

After this PR, users can use a Python data source directly like this:

from pyspark.sql.datasource import DataSource, DataSourceReader

class MyReader(DataSourceReader):
    def read(self, partition):
        yield (0, 1)

class MyDataSource(DataSource):
    @classmethod
    def name(cls):
        return "my-source"

    def schema(self):
        return "id INT, value INT"
    
    def reader(self, schema):
        return MyReader()

spark.dataSource.register(MyDataSource)

df = spark.read.format("my-source").load()
df.show()
+---+-----+
| id|value|
+---+-----+
|  0|    1|
+---+-----+

Why are the changes needed?

To support Python data sources.

Does this PR introduce any user-facing change?

Yes. After this PR, users can load a custom Python data source using spark.read.format(...).load().

How was this patch tested?

New unit tests.

Was this patch authored or co-authored using generative AI tooling?

No

[allisonwang-db]

cc @HyukjinKwon @cloud-fan

[HyukjinKwon]

@cloud-fan @allisonwang-db do we want to support this datasource via USING syntax unlike DSv2, right?

If that's the case, the logics of loading DataSource has to be within DataSource.lookupDataSource and/or DataSource.providingInstance. I don't think we should mix the logics here with DSv2.

[HyukjinKwon]

Let's at least separate the logic into a separate function if possible.

[cloud-fan]

Unfortunately DS v2 TableProvider does not support USING yet. That's why the code is a bit messy here as it's not shared with the SQL USING path. We should support it though...

[HyukjinKwon]

Merged to master.

I had some offline discussion, and I will follow up on my own.

[allisonwang-db]

Thanks @HyukjinKwon!

[HyukjinKwon]

Made a POC (or draft?) PTAL: #43784