Simplify partition structures #763
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fokko
force-pushed
the
fd-simplify-partition-structures
branch
7 times, most recently
from
e988ec3 to
e7ca59c
Compare
This PR removes `SchemalessPartitionSpec` and `UnboundPartitionSpecField`. From the spec: > The field-id property was added for each partition field in v2. > In v1, the reference implementation assigned field ids sequentially > in each spec starting at 1,000. See Partition Evolution for more details. > In v1, partition field IDs were not tracked, but were assigned sequentially > starting at 1000 in the reference implementation. This assignment caused > problems when reading metadata tables based on manifest files from multiple > specs because partition fields with the same ID may contain different data types. > For compatibility with old versions, the following rules are recommended for partition evolution in v1 tables: > - Do not reorder partition fields > - Do not drop partition fields; instead replace the field's transform with the void transform > - Only add partition fields at the end of the previous partition spec I think for simplicity, we should assign the field-IDs starting from 1000, and this will greatly simplify the objects that we need. Next to that, I also believe that users shouldn't have to worry about the field-IDs and that it should be kept internal to Iceberg-Rust.
Fokko
force-pushed
the
fd-simplify-partition-structures
branch
from
e7ca59c to
3eee6e3
Compare
Contributor
Author
|
Closed in favor of #771 |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes
SchemalessPartitionSpecandUnboundPartitionSpecField. We could also combineBoundPartitionSpecandUnboundPartitionSpecif we like, but this is already quite a big change.From the spec:
I think for simplicity, we should assign the field-IDs starting from 1000, and this will greatly simplify the objects that we need. For V1 the
field-IDis missing, and we can just start assigning from 1000 onwards because the IDs are sequential, for V2 tables we deserialize thefield-IDfrom the payload. While I also noticed that we write thefield-idfield for V1 tables in the reference implementation: apache/iceberg#11708Next to that, I also believe that users shouldn't have to worry about the field-IDs and that it should be kept internal to Iceberg-Rust. For the evolution of the partition spec, we should have something similar as Java and PyIceberg, in particular for V1 tables, we have to take the rules above into account, otherwise, there is a serious issue of data-loss, or bricking a table. If we agree on this, I'm happy to implement that API.