feat(datafusion): support metadata tables for Datafusion #879
Conversation
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
d25b534 to
ecf8729
Compare
| [table_name.clone()] | ||
| .into_iter() | ||
| .chain(MetadataTableType::all_types().map(|metadata_table_name| { | ||
| format!("{}${}", table_name.clone(), metadata_table_name.as_str()) |
There was a problem hiding this comment.
The $ syntax follows Flink https://iceberg.apache.org/docs/1.5.2/flink-queries/#inspecting-tables instead of . from Spark, because 4-level catalog.schema.table.metadata is not supported in Datafusion.
| expect![[r#" | ||
| committed_at: PrimitiveArray<Timestamp(Millisecond, Some("+00:00"))> | ||
| [ | ||
| ], | ||
| snapshot_id: PrimitiveArray<Int64> |
There was a problem hiding this comment.
Currently the datafusion test cases don't have any data present so this is empty
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
ecf8729 to
726b0fc
Compare
kevinjqliu
left a comment
kevinjqliu
left a comment
There was a problem hiding this comment.
as a rust rookie, this generally LGTM. glad we're exposing these metadata tables using TableProviders
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
4886b8e to
90abb68
Compare
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
90abb68 to
0b0af5d
Compare
|
Sorry, closed by mistake. |
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
0b0af5d to
88b8b48
Compare
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
2 times, most recently
from
90402b2 to
37a79ad
Compare
Signed-off-by: xxchan <[email protected]>
xxchan
force-pushed
the
xxchan/datafusion-metadata
branch
from
37a79ad to
f830d17
Compare
Signed-off-by: xxchan <[email protected]>
This is resolved (after #1135) and the changes already applied in the PR. This may also be nice to be included in the CLI #1143 😄 |
Signed-off-by: xxchan <[email protected]>
Signed-off-by: xxchan <[email protected]>
Signed-off-by: xxchan <[email protected]>
…etadata Signed-off-by: xxchan <[email protected]>
|
kindly ping @liurenjie1024 @Xuanwo @Fokko |
Signed-off-by: xxchan <[email protected]>
|
kindly ping @liurenjie1024 @Xuanwo @Fokko @sdd |
|
@Xuanwo can we have this merged? 👀 |
I didn't see other concerns so far. Let's move. |
4 participants
Use the newly added metadata tables #823 to support this feature in Datafusion engine to demonstrate its usage.
This may also potentially help design and stablize the interface of metadata tables in the
icebergcrate.