Add support for UV dependency management #2601

geruh · 2025-10-10T17:30:34Z

Rationale for this change

During the release process cleanup, I also had prototyped a migration from Poetry to UV. I used the https://github.com/mkniewallner/migrate-to-uv package to migrate to UV and ran a release workflow and some others like doc serve for validation.

This PR also migrates the Makefile to use UV. I'll still need to upgrade the docs to reflect the UV contribution workflow. I think it would be a good start to push up as a prototype, and have others checkout and give it a try.

Build System Changes

Build Backend

UV doesn't have a Poetry-equivalent build backend with build hooks, so I've switched to setuptools with native Cython support for the avro changes.

[build-system]
requires = ["setuptools>=80", "wheel", "Cython>=3.0.0"]
build-backend = "setuptools.build_meta"

Packaging for release

PEP 639 License Compliance

By going to the test above the license is now properly included in the release metadata

[project]
license = "Apache-2.0"  # SPDX identifier
license-files = ["LICEN[CS]E*", "NOTICE*"]  # Automatic inclusion

Build Verification

Release builds verified:

uv build --sdist  # Source distribution with correct metadata
uv build --wheel  # Wheel with compiled Cython extensions
check-wheel-contents dist/  # Quality checks pass

Wheel vs Poetry:

The migration addresses a packaging issue where Poetry was building non-standard "fat" wheels. Binary comparison using diff revealed significant differences in wheel structure between Poetry and UV builds.

diff -rq uv-wheel poetry-wheel

Only in poetry-wheel/pyiceberg/avro: decoder_basic.c <-- built into .so so not needed :)
Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-310-darwin.so
Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-311-darwin.so
Only in poetry-wheel: NOTICE
Files uv-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so and poetry-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so differ

Poetry's wheels tagged for a specific Python version (e.g., cp312-cp312) contained compiled extensions for multiple Python versions simultaneously. The comparison shows that a Python 3.12 wheel included decoder_fast.cpython-310-darwin.so and decoder_fast.cpython-311-darwin.so alongside the correct binary. These wheels also included source files and placed NOTICE at the wheel root instead of dist-info/licenses/.

The new build process with cibuildwheel creates standard PyPI-compliant wheels where each wheel contains only its corresponding Python version's binary. This reduces wheel size while ensuring proper PEP 639 license file placement in dist-info/licenses/.

Test builds are deployed also can be checked against the repo above.

Are these changes tested?

Yes

make install Environment setup and dependency installation
make test - Unit tests
make test-integration - Integration tests with all extras
make lint - Linting and code quality checks
make doc-install && make doc-serve - Documentation

Are there any user-facing changes?

No user facing changes. Contributors will need to use UV instead of Poetry for development, but all make targets remain the same. So not really?

Fokko

This looks great @geruh, I hear a lot of good stuff around uv 🙌

pyproject.toml

kevinjqliu

Thanks for working on this!

I tested a few things locally.

all the make command
side by side pyproject.toml comparison
ran a few github workflows on my forked repo
- python-ci.yml ✅
- python-ci-docs.yml ✅
- nightly-pypi-build.yml ❌ (something weird with cibuildwheel)

We might want to rebase this PR to capture of of the recent changes, such as the 3.9 deprecation. I'm not too worried about the library versioning since the dependabot workflow will upgrade them again

btw, there are a still references to poetry in mkdocs/docs/contributing.md

pyproject.toml

.github/workflows/svn-build-artifacts.yml

kevinjqliu · 2025-10-26T23:39:47Z

i cant seem to get cibuildwheel to work on github ci 🤔

Error: cibuildwheel: 
Build failed because a pure Python wheel was generated.

If you intend to build a pure-Python wheel, you don't need
cibuildwheel - use `pip wheel .`, `pipx run build --wheel`, `uv
build --wheel`, etc. instead. You only need cibuildwheel if you
have compiled (not Python) code in your wheels making them depend
on the platform.

If you expected a platform wheel, check your project configuration,
or run cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.

kevinjqliu

generally lgtm.
i tested the github workflow changes on my fork

.github/workflows/nightly-pypi-build.yml
.github/workflows/python-release.yml
  .github/workflows/pypi-build-artifacts.yml
  .github/workflows/svn-build-artifacts.yml

.github/workflows/python-ci.yml

.github/workflows/python-ci-docs.yml
.github/workflows/python-release-docs.yml

.github/workflows/pypi-build-artifacts.yml

pyproject.toml

kevinjqliu

I checked all the github workflows, they all run as expected.

I also did a sanity check by comparing the new artifacts (sdist/wheel) against the nightly artifacts thats built with poetry. Noticed some differences

This looks pretty good! Thanks!!

dev/.rat-excludes

pyproject.toml

MANIFEST.in

Co-authored-by: Kevin Liu <[email protected]>

kevinjqliu

LGTM

Check Github Action (https://github.com/kevinjqliu/iceberg-python/actions/runs/19253283756, rest is already checked in CI)
Check artifacts (sdist and wheel) built with fork above
- with sdist, ran RAT check and release process
- with wheel, install locally and verified cpython avro reader (CythonBinaryDecoder) is available
Check all Makefile commands
Check library versions poetry.lock vs uv.lock

Diff sdist and wheel between this branch and nightly from testpypi

sdist

All the changes are as expected
Newly added files are from setuptools, we can try to remove as a followup
Removed files are as expected

> diff -qr ~/Downloads/branch/sdist ~/Downloads/nightly/sdist/ | sort

differ
    Files /Users/kevinliu/Downloads/branch/sdist/Makefile and /Users/kevinliu/Downloads/nightly/sdist/Makefile differ
    Files /Users/kevinliu/Downloads/branch/sdist/PKG-INFO and /Users/kevinliu/Downloads/nightly/sdist/PKG-INFO differ
    Files /Users/kevinliu/Downloads/branch/sdist/dev/.rat-excludes and /Users/kevinliu/Downloads/nightly/sdist/dev/.rat-excludes differ
    Files /Users/kevinliu/Downloads/branch/sdist/pyproject.toml and /Users/kevinliu/Downloads/nightly/sdist/pyproject.toml differ
new
    Only in /Users/kevinliu/Downloads/branch/sdist: MANIFEST.in
    Only in /Users/kevinliu/Downloads/branch/sdist: pyiceberg.egg-info
    Only in /Users/kevinliu/Downloads/branch/sdist: setup.cfg
    Only in /Users/kevinliu/Downloads/branch/sdist: setup.py
removed
    Only in /Users/kevinliu/Downloads/nightly/sdist: build-module.py
    Only in /Users/kevinliu/Downloads/nightly/sdist: poetry.lock

wheel

Changed file is expected, for CythonBinaryDecoder
Newly added file (*dist-info) isnt new, just rename of the old dist-info
Removed files are as expected
- .c, .pyx not needed
- cpython-310, cpython-311 not needed, since this is the 3.12 wheel
- NOTICE file not needed, there's another one in dist-info

> diff -qr ~/Downloads/branch/wheel/ ~/Downloads/nightly/wheel/ | sort

differ
    Files /Users/kevinliu/Downloads/branch/wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so and /Users/kevinliu/Downloads/nightly/wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so differ
new
    Only in /Users/kevinliu/Downloads/branch/wheel: pyiceberg-0.10.0.dev20251111024930.dist-info
removed
    Only in /Users/kevinliu/Downloads/nightly/wheel/pyiceberg/avro: decoder_basic.c
    Only in /Users/kevinliu/Downloads/nightly/wheel/pyiceberg/avro: decoder_fast.cpython-310-darwin.so
    Only in /Users/kevinliu/Downloads/nightly/wheel/pyiceberg/avro: decoder_fast.cpython-311-darwin.so
    Only in /Users/kevinliu/Downloads/nightly/wheel/pyiceberg/avro: decoder_fast.pyx
    Only in /Users/kevinliu/Downloads/nightly/wheel: NOTICE
    Only in /Users/kevinliu/Downloads/nightly/wheel: pyiceberg-0.10.0.dev20251111003004.dist-info

kevinjqliu · 2025-11-11T03:19:29Z

Thanks for the PR @geruh, super exciting to start using uv. Thanks for the review @Fokko @rambleraptor

# Rationale for this change Update contributing documentation to reflect the migration from Poetry to uv for dependency management in #2601. https://docs.astral.sh/uv/ ## Are these changes tested? `make lint && make docs-serve` <img width="625" height="600" alt="image" src="https://github.com/user-attachments/assets/5ff10875-ac2c-432e-b34d-11833968656d" /> ## Are there any user-facing changes? doc changes

# Rationale for this change Follow up to #2601 clean up .egg-info/ and setup.cfg from sdist ## Are these changes tested? Yes Check the new sdist ``` rm -rf dist build pyiceberg.egg-info uv build --sdist tar -tzf dist/pyiceberg-*.tar.gz | grep -E "(build/|setup.cfg|\.egg-info)" ``` Diffed too ``` diff -qr ./dist/pyiceberg-0.10.0 ~/Downloads/nightly/sdist/ | sort differ Files ./dist/pyiceberg-0.10.0/Makefile and /Users/kevinliu/Downloads/nightly/sdist/Makefile differ Files ./dist/pyiceberg-0.10.0/PKG-INFO and /Users/kevinliu/Downloads/nightly/sdist/PKG-INFO differ Files ./dist/pyiceberg-0.10.0/dev/.rat-excludes and /Users/kevinliu/Downloads/nightly/sdist/dev/.rat-excludes differ Files ./dist/pyiceberg-0.10.0/pyproject.toml and /Users/kevinliu/Downloads/nightly/sdist/pyproject.toml differ new Only in ./dist/pyiceberg-0.10.0/tests: .DS_Store Only in ./dist/pyiceberg-0.10.0: MANIFEST.in Only in ./dist/pyiceberg-0.10.0: setup.py removed Only in /Users/kevinliu/Downloads/nightly/sdist: build-module.py Only in /Users/kevinliu/Downloads/nightly/sdist: poetry.lock ``` In sdist, ran RAT check and tests ``` make check-license make test-coverage ``` ## Are there any user-facing changes?  No

# Rationale for this change Follow up to #2601 Apply `PYTHON_ARG` override to all `uv` commands` ## Are these changes tested? Yes Override to use python 3.13 ``` export PYTHON=3.13 make setup-venv make install make docs-serve ``` Unset override and verify it uses system default, python 3.12 ``` unset PYTHON make setup-venv make test ``` ## Are there any user-facing changes?  No

geruh force-pushed the uv branch from c324c0e to f17c28f Compare October 11, 2025 22:51

Fokko approved these changes Oct 12, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

rambleraptor mentioned this pull request Oct 13, 2025

Add pretty-format-toml #2605

Open

kevinjqliu reviewed Oct 26, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

.github/workflows/svn-build-artifacts.yml Outdated Show resolved Hide resolved

kevinjqliu mentioned this pull request Oct 28, 2025

dev: Migrate Python binding dependency management to uv apache/iceberg-rust#1794

Closed

geruh force-pushed the uv branch from f17c28f to d2ef718 Compare October 31, 2025 08:20

kevinjqliu mentioned this pull request Nov 3, 2025

[infra] explore using uv for building wheels #1602

Open

geruh force-pushed the uv branch from d2ef718 to eb3c2b9 Compare November 4, 2025 03:20

geruh mentioned this pull request Nov 4, 2025

infra: use apache/hive:4.0.0 as hive Dockerfile base image #2697

Merged

geruh force-pushed the uv branch from eb3c2b9 to 726f960 Compare November 6, 2025 19:56

kevinjqliu reviewed Nov 6, 2025

View reviewed changes

.github/workflows/pypi-build-artifacts.yml Outdated Show resolved Hide resolved

.github/workflows/pypi-build-artifacts.yml Outdated Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

geruh force-pushed the uv branch from 726f960 to 4ac7e5f Compare November 8, 2025 03:10

kevinjqliu reviewed Nov 10, 2025

View reviewed changes

geruh force-pushed the uv branch from 4d06087 to 4410f7f Compare November 10, 2025 00:34

geruh and others added 10 commits November 10, 2025 17:46

Add support for UV dependency management

bd64217

rebase and fix build

c44231f

fix rat check

cf3f9c7

improve make and skip deprecation of google support

2db841e

Improve build artifacts and test with pypi

0221fe5

add eof space

e1d31f5

explicit includes with manifest.in

cedf02a

include bitmaps in tests

494d5bf

rebase and update based on feedback

22a301b

don't need cython class in sdist!

14309a0

geruh force-pushed the uv branch from f33fcec to 14309a0 Compare November 11, 2025 01:47

Apply suggestion from @kevinjqliu

aa67168

Co-authored-by: Kevin Liu <[email protected]>

kevinjqliu approved these changes Nov 11, 2025

View reviewed changes

kevinjqliu merged commit 75d78c1 into apache:main Nov 11, 2025
8 checks passed

geruh mentioned this pull request Nov 11, 2025

docs: update contributing.md with uv #2733

Merged

This was referenced Nov 11, 2025

sdist: remove .egg-info and setup.cfg from sdist #2734

Merged

Makefile: apply PYTHON_ARG override in all uv commands #2735

Merged

Add support for UV dependency management #2601

Add support for UV dependency management #2601

Uh oh!

Conversation

geruh commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Build System Changes

Build Backend

Packaging for release

Build Verification

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu commented Oct 26, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Diff sdist and wheel between this branch and nightly from testpypi

sdist

wheel

Uh oh!

Uh oh!

kevinjqliu commented Nov 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

geruh commented Oct 10, 2025 •

edited

Loading

kevinjqliu left a comment •

edited

Loading