(chore) use pixi for building

[d-v-b]

Adds pixi for building the project. On my linux desktop, everything compiles with
pixi run pip install -v -e '.[test,test_extras,msgpack,crc32c,pcodec,zfpy]'.

[d-v-b]

not sure if adding the blosc dep here is correct, since we are vendoring it (cc @jakirkham)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.92%. Comparing base (506c89b) to head (4dfd79d).

❌ Your project check has failed because the head coverage (90.92%) is below the target coverage (100.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (506c89b) and HEAD (4dfd79d). Click for more details.

HEAD has 10 uploads less than BASE

Flag	BASE (506c89b)	HEAD (4dfd79d)
	15	5

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #704      +/-   ##
==========================================
- Coverage   99.96%   90.92%   -9.04%     
==========================================
  Files          64       64              
  Lines        2789     2789              
==========================================
- Hits         2788     2536     -252     
- Misses          1      253     +252     

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

cc @mkitti in case you have any advice

[mkitti]

not sure if adding the blosc dep here is correct, since we are vendoring it (cc @jakirkham)

I'll just point out that blosc is not even a dependency of the numcodecs package in conda-forge:
https://github.com/conda-forge/numcodecs-feedstock/blob/main/recipe/meta.yaml

I think this situation is a very weird. We're not even modifying the vendored blosc build here.

In contrast, the blosc package in conda-forge depends on lz4-c, snappy, zlib, and zstd packages, as this package should.
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/meta.yaml

That said, I have no idea how one would have those dependencies in PyPI via wheels.

[martindurant]

cramjam, anyone?? This would just be the reason for them to solidify the blosc backend, but they have all the others in a statically-linked wheel

[mkitti]

statically-linked wheel

Is a big statically-linked wheel the ideal here?

[martindurant]

Is a big statically-linked wheel the ideal here?

If you're worried about dependencies, packaging and how to link to system libs, I suggest that yes. I note that cramjam is ~2MB. Also, zarr likes rust crates, and cramjam can be called directly from rust, and also return python-consumable buffers.

[mkitti]

I took a look at libcramjam. I can already see several major issues. For example, I do not see checksum options for Zstandard. Now in certain circumstances, perhaps this could be fine if we had a plugin system that allowed for multiple implementations of the same codec.

I think I would much rather have a direct dependency on libzstd rather than going through another intermediary.

[mkitti]

Rather than

pixi run pip install -v -e '.[test,test_extras,msgpack,crc32c,pcodec,zfpy]'

I think either pixi install or pixi reinstall should be sufficient. Pixi should be able to handle all the dependencies. If the optional dependencies are desired, they could be inserted via features and (sub)environments.

pixi reinstall is relatively recent.

[cas--]

I noticed the config has an editable install, just be aware there are two open Pixi issues about editable installs causing lockfile churn:

• Remove editable from the lock-file prefix-dev/pixi#1274
• Leave out development package out of lock file prefix-dev/pixi#2512

[d-v-b]

the editable install is not at all required, feel free to amend that if it makes things easier. i'm far from a pixi expert so I'm happy to defer to others here!

[mkitti]

I noticed the config has an editable install, just be aware there are two open Pixi issues about editable installs causing lockfile churn.

Why is this a problem? For a library like this, we could just add the lock file to .gitignore.