Zarr can not handle 0 dimension like numpy does

[will133]

In certain cases, numpy can return a dimension that is 0. This is different from reducing the dimension so the dimension is missing. Therefore, in numpy you can have shape like (1000, 0, 0), which is actually empty data-wise but it is internally consistent. In such case zarr seems to fail:

In [68]: import zarr
In [69]: arr = zarr.empty((1000, 0, 0), dtype='float64')
/.../site-packages/zarr/util.py:48: RuntimeWarning: divide by zero encountered in log10
  target_size = CHUNK_BASE * (2**np.log10(dset_size/(1024.*1024)))

where it would work in numpy:

In [70]: import numpy
In [71]: arr = numpy.empty((1000, 0, 0), dtype='float64')
In [72]: print arr
[]
In [73]: print arr.shape
(1000, 0, 0)

[alimanfoo]

Thanks for raising. Out of interest, could you tell me anything about use cases for zero length dimension?

On Tue, 25 Jul 2017 at 17:21, Will Lee ***@***.***> wrote: In certain cases, numpy can return a dimension that is 0. This is different from reducing the dimension so the dimension is missing. Therefore, in numpy you can have shape like (1000, 0, 0), which is actually empty data-wise but it is internally consistent. In such case zarr seems to fail: In [68]: import zarr In [69]: arr = zarr.empty((1000, 0, 0), dtype='float64') /.../site-packages/zarr/util.py:48: RuntimeWarning: divide by zero encountered in log10 target_size = CHUNK_BASE * (2**np.log10(dset_size/(1024.*1024))) where it would work in numpy: In [70]: import numpy In [71]: arr = numpy.empty((1000, 0, 0), dtype='float64') In [72]: print arr [] In [73]: print arr.shape (1000, 0, 0) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QhwPhpl66ulNuhi-16mkB025O85Nks5sRhXtgaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[will133]

There are cases where you would like to preserve the metadata of the dimension without removing the dimension. For example, if you have an array with 3 dimensions (m, n, p) and you have an index/coordinates for dimension n with values say [1, 3, 9, 10]. When a user slice for index value of 5 on dimension n, instead of an error you can logically end up with an array with shape (m, 0, p). You can think in terms of a flat sql query on a conventional database. In such case it would return an empty table. It's the same when you flatten an n-d array. The difference is that you may still want to keep around the length of the other dimensions.

Thinking in terms of zarr, you can have somebody that produces an empty array through the above slicing operation and the user would like to persist the array on disk with the associated metadata on that dimension. If you reduce that array to nothing, you'll lose all the index info, dimension name, etc.

[will133]

BTW, the reason I was looking at this case is that I had to wrestle with hdf5 to store something like this. Both pytables and h5py does not support this and I believe the underlying hdf5 doesn't support this either. So when we had to persist something like this from numpy we had to jump through some hoops to do so (like writing an 1-d array with size 1 but with metadata saying that the shape is actually (m, 0, p)).

[alimanfoo]

Thanks Will. It may be possible to support this, the error message you posted is actually coming from the code that automatically guesses chunk shape if not specified, it would be interesting to see where Zarr breaks if chunks are explicitly specified. I don't have time to follow up at the moment but happy to discuss further and/or review PR.

…

On Tue, Jul 25, 2017 at 7:41 PM, Will Lee ***@***.***> wrote: BTW, the reason I was looking at this case is that I had to wrestle with hdf5 to store something like this. Both pytables and h5py does not support this and I believe the underlying hdf5 doesn't support this either. So when we had to persist something like this from numpy we had to jump through some hoops to do so (like writing an 1-d array with size 1 but with metadata saying that the shape is actually (m, 0, p)). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150#issuecomment-317831025>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QgfNE3xfJUhKwFhyIoFcIfBIpUTAks5sRjbQgaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[newt0311]

I'm affected by this as well. There are a number of cases where it is convenient to create a zero-length array and store it just to preserve the metadata (later processes can fill it in).

I'll try to take a look at this if I get a chance.

[jakirkham]

Also running into this issue. Wanting this functionality for similar reasons already outlined. Though would add that I try to collect a group of objects and may find any number of objects (including 0), but still want to be able to store and handle the result the same for the 0 case.

Tried specifying the chunks when creating the Zarr Dataset, which seemed to work. However it doesn't appear to be possible to retrieve the data. This includes trying to simply access the Zarr Dataset without loading the data into an array. So it seems there are other issues that would need to be addressed beyond fixing automatic chunking.

[alimanfoo]

Hi @jakirkham, could you say a bit more about how you would expect a zarr array with one or more zero length dimensions to behave why you try to access data?

…

On Thu, Aug 31, 2017 at 4:16 PM, jakirkham ***@***.***> wrote: Also running into this issue. Wanting this functionality for similar reasons already outlined. Though would add that I try to collect a group of objects and may find any number of objects (including 0), but still want to be able to store and handle the result the same for the 0 case. Tried specifying the chunks when creating the Zarr Dataset, which seemed to work. However it doesn't appear to be possible to retrieve the data. This includes trying to simply access the Zarr Dataset without loading the data into an array. So it seems there are other issues that would need to be addressed beyond fixing normalization of chunks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150#issuecomment-326328140>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qj5lQ7HRdapMT3qPw-tBnDY556Jwks5sds5YgaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[jakirkham]

So I can provide an example of how h5py behaves in a similar case. Would expect Zarr to behave the same.

In [1]: import numpy as np

In [2]: import h5py as h5

In [3]: a = np.zeros((1, 2, 3), dtype=np.uint8)

In [4]: b = np.zeros((0, 2, 3), dtype=np.uint8)

In [5]: with h5.File("test.h5", "w") as fh:
   ...:     fh["a"] = a
   ...:     fh["b"] = b
   ...:     

In [6]: with h5.File("test.h5", "r") as fh:
   ...:     for dn in ["a", "b"]:
   ...:         print(fh[dn])
   ...:         print(fh[dn][...])
   ...:         print(type(fh[dn][...]))
   ...:         print("")
   ...:         
<HDF5 dataset "a": shape (1, 2, 3), type "|u1">
[[[0 0 0]
  [0 0 0]]]
<class 'numpy.ndarray'>

<HDF5 dataset "b": shape (0, 2, 3), type "|u1">
[]
<class 'numpy.ndarray'>

Regarding why one would want to access or store data with 0 dimensions, it is basically about avoiding special casing. If I don't know whether or not a dimension will be length 0, then I shouldn't need to check before storing or loading such data. It makes it much more difficult to reason about and test code that requires this sort of special casing.

[alimanfoo]

Thank you, very helpful.

…

On Thu, Aug 31, 2017 at 6:56 PM, jakirkham ***@***.***> wrote: So I can provide an example of how h5py behaves in a similar case. Would expect Zarr to behave the same. In [1]: import numpy as np In [2]: import h5py as h5 In [3]: a = np.zeros((1, 2, 3), dtype=np.uint8) In [4]: b = np.zeros((0, 2, 3), dtype=np.uint8) In [5]: with h5.File("test.h5", "w") as fh: ...: fh["a"] = a ...: fh["b"] = b ...: In [6]: with h5.File("test.h5", "r") as fh: ...: for dn in ["a", "b"]: ...: print(fh[dn]) ...: print(fh[dn][...]) ...: print(type(fh[dn][...])) ...: print("") ...: <HDF5 dataset "a": shape (1, 2, 3), type "|u1"> [[[0 0 0] [0 0 0]]]<class 'numpy.ndarray'> <HDF5 dataset "b": shape (0, 2, 3), type "|u1"> []<class 'numpy.ndarray'> Regarding why one would want to access or store data with 0 dimensions, it is basically about avoiding special casing. If I don't know whether or not a dimension will be length 0, then I shouldn't need to check before storing or loading such data. It makes it much more difficult to reason about and test code that requires this sort of special casing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150#issuecomment-326374202>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QpETcchsd40xP-aOVeo-HtRO3dGxks5sdvO-gaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[newt0311]

FYI: I submitted a pull request to try to fix this. I thought you might find it to be useful...

-- PG.

[alimanfoo]

Thank you, apologies I've been tied up on other projects, I'll take a look asap.

…

On Monday, September 18, 2017, Prakhar Goel ***@***.***> wrote: FYI: I submitted a pull request to try to fix this. I thought you might find it to be useful... -- PG. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150#issuecomment-330188736>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qr0aZHo0RTjQDBsUn02ewaa2fZXqks5sjk7kgaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[rabernat]

Just a heads up that we hit this in the zarr backend for xarray (pydata/xarray#1528). Zero-dimensional arrays (i.e. just single values) are supported in netCDF, and in numpy we can do

ar = np.empty(shape=())
ar[...] = 0

So we would ideally like to be able to do the following in zarr

za = zarr.create(shape=(), store='tmp_file')
za[...] = 0

Currently this raises a permission error.

(edited per @shoyer's comment below)

[alimanfoo]

I'm currently on leave but happy to work on this when I'm back in a couple of weeks and/or review PR. FWIW there may be different code paths needed to handle an array with 1 or more dimensions of zero length versus an array with no dimensions (single value).

On Sun, 8 Oct 2017 at 05:27, Ryan Abernathey ***@***.***> wrote: Just a heads up that we hit this in the zarr backend for xarray ( pydata/xarray#1528 <pydata/xarray#1528>). Zero-dimensional arrays (i.e. just single values) are supported in netCDF, so we would ideally like to be able to do za = zarr.create(shape=(), store='tmp_file') za[:] = 0 Currently this raises a permission error. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/150#issuecomment-334982273>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QrAEltGfGBlUa6HrTp3JEGQQag2Aks5sqE8dgaJpZM4Oiynu> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo