Failures on Python 3.10

[vstinner]

Hi,

39 tests fails when run on Python 3.10. See logs below. There are 4 different errors:

• _lib.pymdb_put(): TypeError: initializer for ctype 'char *' must be a cdata pointer, not bytearray
• _lib.pymdb_put(): TypeError: initializer for ctype 'char *' must be a cdata pointer, not numpy.ndarray
• _lib.pymdb_put(): TypeError: len() of unsized object
• _lmdb_decode_key_buffer(): AttributeError: '_cffi_backend.buffer' object has no attribute 'tobytes'

I tested zarr-python 2.5.0. I get similar the same errors with zarr-python 2.6.1.

Downstream Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1903075

Full tests log: tests.log

Version and installation information

Please provide the following:

• Value of zarr.__version__: 2.5.0
• Value of numcodecs.__version__: 0.7.2
• Version of Python interpreter: Python 3.10.0a2
• Operating system (Linux/Windows/Mac): Linux
• How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): build the Fedora package, see https://src.fedoraproject.org/rpms/python-zarr/tree/master

[vstinner]

_lib.pymdb_put(): TypeError: len() of unsized object

This one comes from: zarr/tests/test_core.py::TestArrayWithLMDBStore::test_array_0d

The error comes from len(value) where value type is: <class 'numpy.ndarray'>.

On the tested numpy 1.19.4, 0d arrays have no length. Example:

$ python3
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
>>> import numpy
>>> x = numpy.array(1.0, numpy.float64)
>>> x
array(1.)
>>> numpy.ndim(x)
0
>>> len(x)
TypeError: len() of unsized object

[vstinner]

_lmdb_decode_key_buffer(): AttributeError: '_cffi_backend.buffer' object has no attribute 'tobytes'

_cffi_backend.buffer never had a tobytes() method. Maybe previously, the key type was not _cffi_backend.buffer but another type?

Maybe key[:] or bytes(buffer) can be used instead.

I don't understand what is the "ffi" module in the CFFI documentation. I found ffi.buffer documentation, not sure if it's related:
https://cffi.readthedocs.io/en/latest/ref.html#ffi-buffer-ffi-from-buffer

At least, this test pass with this change:

--- zarr/storage.py.old	2020-12-03 15:14:45.162461403 +0100
+++ zarr/storage.py	2020-12-03 15:14:57.713510614 +0100
@@ -1763,7 +1763,7 @@
 
 def _lmdb_decode_key_buffer(key):
     # assume buffers=True
-    return key.tobytes().decode('ascii')
+    return bytes(key).decode('ascii')
 
 
 def _lmdb_decode_key_bytes(key):

[vstinner]

_lib.pymdb_put(): TypeError: initializer for ctype 'char *' must be a cdata pointer, not bytearray
_lib.pymdb_put(): TypeError: initializer for ctype 'char *' must be a cdata pointer, not numpy.ndarray

This function comes from the lmdb project and the pymdb_put() function is defined with cffi as:

    static int pymdb_put(MDB_txn *txn, MDB_dbi dbi, char *key_s, size_t keylen,
                         char *val_s, size_t vallen, unsigned int flags)
    {
        MDB_val key = {keylen, key_s};
        MDB_val val = {vallen, val_s};
        return mdb_put(txn, dbi, &key, &val, flags);
    }

I understand that previously, bytearray and numpy.ndarray types were accepted by cffi as char*, but it's no longer the case.

[vstinner]

I tested cffi version 1.14.3 (Fedora package python3-cffi-1.14.3-1.fc34.x86_64).

[Carreau]

My guess is this directly affect https://github.com/jnwatson/py-lmdb/ or on PyPI: https://pypi.org/project/lmdband not zarr which is pure python.

Which version of PyLMDB ?

Do you want me to forward this to above repository or do you wish to do that yourself to have less indirections ?

[Carreau]

(replied on downstream bug)

[jakirkham]

I think what Matthias is saying is true. Though maybe we can make some changes of our own to be less sensitive to LMDB's behavior. Submitted PR ( #670 ) to do that. This also puts the LMDBStore more inline with how we handle other similar backend storage issues.

[jakirkham]

@vstinner, could you please try that PR and let us know how it goes?

[vstinner]

Downstream Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1903075

The zarr test suites pass on Python 3.10 after @encukou upgraded lmdb from 0.92 to 1.0.0. So we don't need to test #670 on Fedora. Thanks for your quick feedback, the issue is now solved on our side!

[jakirkham]

Thanks for the update!