feat: add user defined table function support (#1113)

* Initial commit of udtf work

* Add table provider capsule to PyTableProvider

* working through functions to register udtf

* Moving ffi-table-provider to datafusion-ffi-example since we will cover more examples than just table providers

* Update example naming and split into different files for table provider and table function

* Update udtf to handle python based table functions

* Add python table function example and test

* Update CI workflow for new file location

* Add documentation

* Add table decorator and unit test

* pin arrow to 55.0.0

* Allow large error in order to match datafusion trait this is called for

* Ruff formatting

Author: timsaucer

.github/workflows/test.yaml

@@ -91,9 +91,9 @@ jobs:
 
       - name: FFI unit tests
         run: |
-          cd examples/ffi-table-provider
+          cd examples/datafusion-ffi-example
           uv run --no-project maturin develop --uv
-          uv run --no-project pytest python/tests/_test_table_provider.py
+          uv run --no-project pytest python/tests/_test*.py
 
       - name: Cache the generated dataset
         id: cache-tpch-dataset

docs/source/user-guide/common-operations/udf-and-udfa.rst

@@ -242,3 +242,35 @@ determine which evaluate functions are called.
     })
 
     df.select("a", exp_smooth(col("a")).alias("smooth_a")).show()
+
+Table Functions
+---------------
+
+User Defined Table Functions are slightly different than the other functions
+described here. These functions take any number of `Expr` arguments, but only
+literal expressions are supported. Table functions must return a Table
+Provider as described in the ref:`_io_custom_table_provider` page.
+
+Once you have a table function, you can register it with the session context
+by using :py:func:`datafusion.context.SessionContext.register_udtf`.
+
+There are examples of both rust backed and python based table functions in the
+examples folder of the repository. If you have a rust backed table function
+that you wish to expose via PyO3, you need to expose it as a ``PyCapsule``.
+
+.. code-block:: rust
+
+    #[pymethods]
+    impl MyTableFunction {
+        fn __datafusion_table_function__<'py>(
+            &self,
+            py: Python<'py>,
+        ) -> PyResult<Bound<'py, PyCapsule>> {
+            let name = cr"datafusion_table_function".into();
+
+            let func = self.clone();
+            let provider = FFI_TableFunction::new(Arc::new(func), None);
+
+            PyCapsule::new(py, provider, Some(name))
+        }
+    }