Arrow: Set field-id with prefix #227
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While working on the write support, I started to understand the meaning of the Arrow metadata. The `PARQUET:` prefix means that it is specific to Parquet, and setting `PARQUET:field_id` will actually not be set as metadata, but as the `field_id`. I think we should remove the other option without the prefix. I also did the same for the doc. I checked, in Java we don't check for the documentation, but I think it is a nice addition. However, I think we should only check for the key that's the same as the Iceberg field (`doc`).
HonahX
approved these changes
Dec 21, 2023
Contributor
HonahX
left a comment
HonahX
left a comment
There was a problem hiding this comment.
LGTM! Just have one small comment
sungwy
approved these changes
Dec 21, 2023
Merged
sungwy
pushed a commit
to sungwy/iceberg-python
that referenced
this pull request
Jan 13, 2024
'PARQUET:' prefix is specific to Parquet, with 'PARQUET:field_id' setting the 'field_id'. Removed the non-prefixed alternative for 'field_id'. Removed the prefixed alternative for 'doc'.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on the write support, I started to understand the meaning of the Arrow metadata. The
PARQUET:prefix means that it is specific to Parquet, and settingPARQUET:field_idwill actually not be set as metadata, but as thefield_id. I think we should remove the other option without the prefix.I also did the same for the doc. I checked, in Java we don't check for the documentation, but I think it is a nice addition. However, I think we should only check for the key that's the same as the Iceberg field (
doc).