test: Add Some Spark Client Tests and Update Documentation on Generic Tables #3000
Conversation
adam-christian-software
changed the title
|
@gh-yzou & @flyrain - Here's the docs update for limitation around credential vending: https://apache-polaris.slack.com/archives/C084QSKD6S2/p1762453063740029?thread_ts=1762203273.837449&cid=C084QSKD6S2 I know that @gh-yzou did a good job at answering Abed's question, so potentially, we can build on this with Yun's answers there. |
| --conf spark.sql.sources.useV1SourceList='' | ||
| ``` | ||
|
|
||
| # Limitations |
There was a problem hiding this comment.
how about update this title to # Current Limitations, because all those limitations can be eventually removed, just require extra work
| 2. Generic tables (non-Iceberg tables) do not currently support credential vending. | ||
|
|
||
| ## Delta Lake Limitations | ||
| 1. Create table as select (CTAS) is not supported for Delta Lake tables. As a result, the `saveAsTable` method of `Dataframe` |
There was a problem hiding this comment.
I see we updated Delta tables to Delta Lake Tables. I think Delta is the actual table metadata format, Delta lake seems more indicating the storage layer or system, and within delta lake, the table is stored in delta format. So from table format's point of view, I think Delta table is a more accurate term.
| --- | ||
|
|
||
| The Generic Table in Apache Polaris is designed to provide support for non-Iceberg tables across different table formats includes delta, csv etc. It currently provides the following capabilities: | ||
| The generic tables framework provides support for non-Iceberg table formats including Delta Lake, CSV, etc. With this framework, you can: |
There was a problem hiding this comment.
I don't think generic table is a framework, it is a catalog concept, can we keep what we originally have?
| 2) No commit coordination or update capability provided at the catalog service level. | ||
|
|
||
| Therefore, the catalog itself is unaware of anything about the underlying table except some of the loosely defined metadata. | ||
| It is the responsibility of the engine (and plugins used by the engine) to determine exactly how loading or committing data |
There was a problem hiding this comment.
This part explains the current contract between the server and client, can we add this part back?
| The Spark client can manage Iceberg tables and non-Iceberg tables. | ||
|
|
||
| Note the Polaris Spark client is able to handle both Iceberg and Delta tables, not just Delta. | ||
| Users who only use Iceberg tables can use Spark without this client. |
There was a problem hiding this comment.
-> For users who only need to interact with iceberg tables is not strictly required to use Polaris Spark Client. Regular Iceberg provided Spark Client should continue work.
2 participants
Context
As I was investigating how to support Parquet files using Generic Tables, I performed a few miscellaneous clean-ups:
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)Docs Pictures