Expose Avro reader to PyIceberg

[Fokko]

Which issue does this PR close?

I've been looking into exposing the Avro readers to PyIceberg. This will give a huge benefit to PyIceberg because we can drop the Cython Avro reader.

What changes are included in this PR?

Exposing methods and structures to read the manifest lists, and manifests itself.

Are these changes tested?

By using them in PyIceberg :)

[sdd]

Do you have an example of the JSON input that fails deserialization into a StructType? If so I'll see what I can do

[Fokko]

Thanks @sdd for jumping in here 👍

I would expect the following to work:

Suggested change

	let res: Result<Type, _> = serde_json::from_str(json);
	let res = serde_json::from_str<StructType>(json);

I was also able to reproduce this in a unit test:

#[test]
fn empty_struct_type() {
    let json = r#"{"type": "struct", "fields": []}"#;

    let expected = StructType {
        fields: vec![],
        id_lookup: OnceLock::default(),
        name_lookup: OnceLock::default(),
    };

    let res = serde_json::from_str::<StructType>(json).unwrap();

    assert_eq!(res, expected);
}

But it looks like we need to wrap it in the Type enum.

[Xuanwo]

Hi @Fokko, I experimented a bit with this PR. One possible approach is to allow Python to access our structs in _serde, which map directly to the on-disk representation without any type transformation or parsing.

We could have something like this:

#[pyfunction]
pub fn read_manifest_list_v2(bs: &[u8]) -> PyManifestList {
    let reader = apache_avro::Reader::new(bs).unwrap();
    let values = apache_avro::types::Value::Array(
        reader
            .collect::<std::result::Result<Vec<apache_avro::types::Value>, _>>()
            .unwrap(),
    );
    let manifest_list = apache_avro::from_value::<_serde::ManifestListV2>(&values).unwrap();

    PyManifestList {
        inner: manifest_list,
    }
}

Or much better if we can expose such API directly:

#[pyfunction]
pub fn read_manifest_list_v2(bs: &[u8]) -> PyManifestList {
    PyManifestList {
        inner: ManifestList::parse_as_is(bs),
    }
}

Our current design focuses solely on Rust users, but some users may simply want to parse the file themselves and don’t want iceberg-rust to handle any transformation (such as parsing into Datum).

We could reconsider this, perhaps we can expose these as a public API, but hide them behind a feature gate.

cc @liurenjie1024 and @sdd for ideas.

[Fokko]

Our current design focuses solely on Rust users, but some users may simply want to parse the file themselves and don’t want iceberg-rust to handle any transformation (such as parsing into Datum).

Yes, that makes sense to me. I think we still want to have Iceberg-Rust some things like setting the default values for V2 (eg, setting 134: content to data, when reading V1 metadata):

Apart from that, I think your approach is great. Curious to learn what others think.

[liurenjie1024]

Should we consider having a values.rs module like what we did in core crate?

[liurenjie1024]

Our current design focuses solely on Rust users, but some users may simply want to parse the file themselves and don’t want iceberg-rust to handle any transformation (such as parsing into Datum).

I'm leaning toward to this approach, also this makes the api more aligned with python/java implementation.

[Fokko]

Thanks everyone for chiming in here. Let me summarize the discussion. I think there is consensus that the callback is not ideal.

1. Supply required information to construct the summaries
   1. Instead of having the Fn(i32) -> Result<Option<StructType>> provider, we could pass in a HashMap<i32, StructType>. We would bind all the PartitionSpec's in PyIceberg. This is relative straightforward, but comes at a cost when there are many PartitionSpecs (which should be okay for the majority of tables).
   2. What @kevinjqliu suggested Expose Avro reader to PyIceberg #1328 (comment) suggested. Pass in the current Schema and PartitionSpec's to Iceberg-Rust where we can do the lazy binding on the Iceberg-Rust side.
   3. Go all the way, and convert the TableMetadata to Iceberg-Rust, this is probably where we end up at some point at some day, but require a lot of scaffolding.
2. Deserialize in Vec<u8> instead of a Datum, and convert them later into the actual type. This removes the dependency on the Schema and the PartitionSpec's.

I'm leaning towards 2 since that aligns the best with PyIceberg, where we can deserialize the manifest-list without having to know about the schema. I would make sure that we have consensus before moving into a certain direction, and happy to follow up on that.

[liurenjie1024]

Thanks everyone for chiming in here. Let me summarize the discussion. I think there is consensus that the callback is not ideal.

1. Supply required information to construct the summaries

   1. Instead of having the Fn(i32) -> Result<Option<StructType>> provider, we could pass in a HashMap<i32, StructType>. We would bind all the PartitionSpec's in PyIceberg. This is relative straightforward, but comes at a cost when there are many PartitionSpecs (which should be okay for the majority of tables).
   2. What @kevinjqliu suggested Expose Avro reader to PyIceberg #1328 (comment) suggested. Pass in the current Schema and PartitionSpec's to Iceberg-Rust where we can do the lazy binding on the Iceberg-Rust side.
   3. Go all the way, and convert the TableMetadata to Iceberg-Rust, this is probably where we end up at some point at some day, but require a lot of scaffolding.

2. Deserialize in Vec<u8> instead of a Datum, and convert them later into the actual type. This removes the dependency on the Schema and the PartitionSpec's.

I'm leaning towards 2 since that aligns the best with PyIceberg, where we can deserialize the manifest-list without having to know about the schema. I would make sure that we have consensus before moving into a certain direction, and happy to follow up on that.

+1

[kevinjqliu]

I like #2 as well. The refactor should be less effort than scaffolding between python class and rust struct

[Fokko]

Thanks for chiming in here, I've created PR #1369 that implements #2. PTAL

[Xuanwo]

#1369 has been merged, maybe we can remove the callback now.

[Fokko]

@Xuanwo @liurenjie1024 @kevinjqliu This is ready for another round of reviews :)

[kevinjqliu]

I pushed a fix for CI running on Windows OS

The proper fix requires fixing how path is handled by pyiceberg's PyArrowFileIO class. This is an issue on the pyiceberg side too when running tests/utils/test_manifest.py::test_read_manifest_entry on Windows. I'll open an issue on the PyIceberg side with more details

[Fokko]

@kevinjqliu Thanks for pushing the fix! I would expect that Windows will still work against an object store, but it cannot handle the c:/ prefix.

[Fokko]

@Xuanwo @liurenjie1024 CI is now green 💚

[Xuanwo]

Thank you for working on this, let's move!