partitioned write support

[jqin61]

Todo

• support partitioned append()
  • support append with identity transform
  • fix scenario when arrow table schema not aligned with iceberg schema (finished by others)
  • add integration test for null column partitioning after issue#348 is closed
  • avoid sorting input arrow table when it is already sorted
• support partitition field in manifest file see PartitionKey
• apply transform for partitioning algorithm efficiency analysis when transform involved
• support partitioned static overwrite()
  • overwrite entire table
  • overwrite with expression or filter string (specified partition)
  • overwrite filter validatoin (validation for static overwrite with filter jqin61/iceberg-python#4) as discussed in the monthly meeting, overwrite will be supported by delete + append. we will support more wild filters than spark iceberg and might rewrite files for overwriting rather than just using IsNull and EqualTo. So this is not needed.
• extend summary for partitioned stats (Add partition stats in snapshot summary #521)
• support partitioned dynamic overwrite()

[sungwy]

Suggested change

	def get_partition_sort_order(partition_columns: list[str], reverse: bool = False) -> dict[str, Any]:
	def _get_partition_sort_order(partition_columns: list[str], reverse: bool = False) -> dict[str, Any]:

This is a nit: should we use _single_leading_underscore to indicate that these are internal functions according to PEP-8?

[jqin61]

Yes, thanks for the reminder! More refactoring is on the way.

[jqin61]

Default value is True, which breaks the avro writer encoding manifest entry when the partition value is null. (Because the attribute of required being True in the nested field of the Avro schema will be used by the writer to infer and choose a non-optional field writer that could not encode None value.)

This is for encoding only, so set it to False for now.

TODO: Set it properly according to table schema.

[jqin61]

As discussed in the monthly community sync, this will be broken down into 4 prs of:

1. Partitioned append with identity transform
2. Dynamic overwrite using delete + append, 2 snapshots in one commit
3. Hidden partitioning support (for slicing the arrow table, manifest file entry.partition, data file path)
4. Static overwrite using delete + append, 2 snapshots in one commit

[kevinjqliu]

hey @sungwy @jqin61 just wanted to double check that this PR is no longer relevant. I believe all components of partitioned write support has been already merged