Need to use field.name to determine arrow field's position when PARQUET:field_id is unavailable

[CTTY]

When using ParquetWriter to write RecordBatch and it's counting nan values, it will need to walk through both RecordBatch's schema and Iceberg schema in a partner fashion:

iceberg-rust/crates/iceberg/src/writer/file_writer/parquet_writer.rs

Line 528 in 9787140

.compute(self.schema.clone(), batch_c)?;

Basically the call stack is NanValueCountVisitor::compute -> visit_struct_with_partner -> ArrowArrayAccessor::field_partner -> get_field_id

This will fail in get_field_id when the incoming arrow schema doesn't have PARQUET:field_id, which is the case for inserting data using DataFusion.

Originally posted by @CTTY in #1511 (comment)

[liurenjie1024]

Hi, @CTTY Thanks for raising this issue. Instead of using field.name to match field, I would propose to change ArrowArrayAccessor to accept an ArrowSchema, which is converted from ParquetWriter 's schema. And ArrowArrayAccessor should use passed in ArrowSchema to match.

The motivation behind this is that the array schema is passed by user, and it's not quite friendly to ask user to match their name, WDYT?

[CTTY]

Hi @liurenjie1024 , thanks for the suggestion! However, I'm not sure how passing ArrowSchema to ArrowArrayAccessor can free us from matching fields via field.name. Could you please elaborate more?

My understanding is that even if we convert the schema to ArrowSchema, the schema still won't have PARQUET:field_id for the insert_into case, and we will need to match using field.name

[liurenjie1024]

Hi @liurenjie1024 , thanks for the suggestion! However, I'm not sure how passing ArrowSchema to ArrowArrayAccessor can free us from matching fields via field.name. Could you please elaborate more?

My understanding is that even if we convert the schema to ArrowSchema, the schema still won't have PARQUET:field_id for the insert_into case, and we will need to match using field.name

Hi, @CTTY When you convert iceberg schema to arrow schema, we will insert the PARQUET:field_id metadata, see

iceberg-rust/crates/iceberg/src/arrow/schema.rs

Line 466 in fbc3716

(PARQUET_FIELD_ID_META_KEY.to_string(), field.id.to_string()),

.

But I'm questioning the necessity of id matching or even name matching in this case. From a user's point of view, they just need to ensure that the passed in arrow array matches iceberg's schema, e.g. type matches. They don't need to care about name or ids. I think a better approach is to match array just by order and array type? For example, when the writer's iceberg schema is following:

{
    id int,
    name string,
    address string
}

The user is expected to pass record batches of three arrays:

int array
string array
address array

They should follow order of iceberg schema, and type should matches. This requirements seems more user friendly to me, what do you think?

[Fokko]

I think a better approach is to match array just by order and array type?

I think this is typically the approach for SQL:

spark-sql (default)> CREATE TABLE vo (a string, b int);
Time taken: 0.516 seconds

spark-sql (default)> insert into vo SELECT 1 as b, 'one' as a;
25/08/04 12:16:51 WARN TableOutputResolver: The query columns and the table columns have same names but different orders. You can use INSERT [INTO | OVERWRITE] BY NAME to reorder the query columns to align with the table columns.
[INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST] Cannot write incompatible data for the table `spark_catalog`.`default`.`vo`: Cannot safely cast `b` "STRING" to "INT". SQLSTATE: KD000

spark-sql (default)> insert into vo VALUES ('one', 1);
Time taken: 0.895 seconds

If we want to map names to field-IDs, we should use name-mapping: https://iceberg.apache.org/spec/#column-projection

[CTTY]

Hi @liurenjie1024 and @Fokko , thanks for the inputs! I've updated my PR to fall back to name mapping using the ArrowSchema converted from Iceberg schema, ptal