feat: support delete if empty for parquet writer #838
Conversation
|
I haven't add the test for this PR yet because I am not sure whether it's a reasonable design to provide the delete function here. But in practice, it's useful I think. Welcome to any suggestions. cc @liurenjie1024 @Xuanwo |
ZENOTME
force-pushed
the
delete_empty_row_writer
branch
from
0a4844e to
4e07518
Compare
|
I think this PR is ready to review. cc @Xuanwo @liurenjie1024 |
liurenjie1024
left a comment
liurenjie1024
left a comment
There was a problem hiding this comment.
Thanks @ZENOTME for this pr, instead of adding an extra parameter, should we consider making opening the file lazily, e.g. only create one when we actually append data?
crates/iceberg/src/io/file_io.rs
Outdated
| } | ||
|
|
||
| /// Delete the file if it exists. | ||
| pub async fn delete(&self) -> crate::Result<()> { |
There was a problem hiding this comment.
I think this should be a function of FileIO
Thanks @liurenjie1024. I think it's a good idea. Let me try this |
ZENOTME
force-pushed
the
delete_empty_row_writer
branch
from
4e07518 to
b928c61
Compare
|
Hi @liurenjie1024, I use the lazy init to avoid creating new file when empty write. PTAL |
liurenjie1024
left a comment
liurenjie1024
left a comment
There was a problem hiding this comment.
Thanks @ZENOTME for fixing this, LGTM!
2 participants
This PR lets the user indicate to delete the file if the writer doesn't write anything.