The iceberg-datafusion Table Provider has a way to scan the latest snapshots dynamically

[phillipleblanc]

Is your feature request related to a problem or challenge?

Currently the DataFusion IcebergTableProvider takes a fixed table metadata on creation, and data that is added to the Iceberg table after that is not reflected in the scans.

Describe the solution you'd like

Expose the IcebergTableProvider::try_new that takes in the catalog, and use the catalog on each table scan to get the latest snapshot of the table. If the table is constructed with the existing IcebergTableProvider::try_new_with_table or IcebergTableProvider::try_new_from_table_snapshot methods, then it should continue to work as is on that static snapshot.

Willingness to contribute

I can contribute to this feature independently

[CTTY]

Some related discussion here