Challenges running xarray wrapped netcdf files

[pwolfram]

This is a traceback from calling compute on an XArray computation on dask.distributed.

We're able to use dask.array on a NetCDF4 object without locks if our workers have single threads. However, when computing on the .data attribute backed by a NetCDF object wrapped by a few XArray containers we run into the following error. It appears to be coming from computing the shape, which is odd. Traceback below:

cc @mrocklin @shoyer

In [168]: ds = xr.open_mfdataset(fname, lock=False)

In [169]: ds.yParticle.data.sum().compute()

/net/scratch3/pwolfram/miniconda2/lib/python2.7/site-packages/dask/array/core.pyc in getarray()
     47         lock.acquire()
     48     try:
---> 49         c = a[b]
     50         if type(c) != np.ndarray:
     51             c = np.asarray(c)

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in __getitem__()
    396 
    397     def __getitem__(self, key):
--> 398         return type(self)(self.array, self._updated_key(key))
    399 
    400     def __setitem__(self, key, value):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in _updated_key()
    372 
    373     def _updated_key(self, new_key):
--> 374         new_key = iter(canonicalize_indexer(new_key, self.ndim))
    375         key = []
    376         for size, k in zip(self.array.shape, self.key):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in ndim()
    380     @property
    381     def ndim(self):
--> 382         return len(self.shape)
    383 
    384     @property

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
    384     def shape(self):
    385         shape = []
--> 386         for size, k in zip(self.array.shape, self.key):
    387             if isinstance(k, slice):
    388                 shape.append(len(range(*k.indices(size))))

/users/pwolfram/lib/python2.7/site-packages/xarray/conventions.pyc in shape()
    447     @property
    448     def shape(self):
--> 449         return self.array.shape[:-1]
    450 
    451     def __str__(self):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
    384     def shape(self):
    385         shape = []
--> 386         for size, k in zip(self.array.shape, self.key):
    387             if isinstance(k, slice):
    388                 shape.append(len(range(*k.indices(size))))

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in shape()
    407     @property
    408     def shape(self):
--> 409         return self.array.shape
    410 
    411     def __array__(self, dtype=None):

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:32778)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._getdims (netCDF4/_netCDF4.c:31870)()

RuntimeError: NetCDF: Not a valid ID

[mrocklin]

I worked with @pwolfram to produce that traceback. We were able to get things working if we dove into the .dask graph to get the NetCDF file and then wrap it with a dask.array, something like the following:

xr_var = ds.yParticle.data.dask['some-key']
var = xr_var.array.array.array # unpack

x = da.from_array(var, chunks=(...))
x.sum().compute()  # works ok

[shoyer]

This makes sense now that I'm looking at the full traceback.

This making use of conventions.CharToStringArray, another array-like type that xarray uses for decoding string data, for which we are relying on default serialization (pickle). The netCDF4.Variable get unpickled an no longer points to an open file.

Let's try this again opening the dataset with decode_cf=False.

[pwolfram]

It looks the same.

[shoyer]

Somehow we seem to be using a CharToStringArray....

On Sun, Nov 6, 2016 at 2:33 PM, Phillip Wolfram notifications@github.com
wrote:

It looks the same.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#629 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKS1stI69DmsWS6PvJ_rjuZMoWRurNxks5q7lWngaJpZM4Kqsny
.

[shoyer]

What are the types of each level of xr_var.array.array.array?

Also try concat_characters=False

[shoyer]

Also: xr.open_dataset(fname, lock=False, chunks={}) should make a dask array without the mulit-file logic.

[pwolfram]

concat_characters=False is the same error.

[pwolfram]

levels of xr_var.array.array.array:

In [204]: type(xr_var.array)
Out[204]: xarray.core.indexing.LazilyIndexedArray

In [205]: type(xr_var.array.array)
Out[205]: xarray.backends.netCDF4_.NetCDF4ArrayWrapper

In [206]: type(xr_var.array.array.array)
Out[206]: netCDF4._netCDF4.Variable

[pwolfram]

xr.open_dataset(fname, lock=False, chunks={}) is the same error.

[shoyer]

Following your stacktrace down, I see five array types:

LazilyIndexedArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
--> 386         for size, k in zip(self.array.shape, self.key):

CharToStringArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/conventions.pyc in shape()
--> 449         return self.array.shape[:-1]

LazilyIndexedArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
--> 386         for size, k in zip(self.array.shape, self.key):

NetCDF4ArrayWrapper (most likely, definitely a NDArrayMixin subclass):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in shape()
--> 409         return self.array.shape

netCDF4.Variable:

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:32778)()

So it really looks like dask-distributed is choking on CharToStringArray.

In particular, it appears that the original error is from a different array than xr_var. Are you sure you're pulling out the top level key from the dask dict?

To be doubly sure, what is the dtype of ds.yParticle? It is also conceivable (though I think unlikely) that some other array with string data has been pulled into it's dask graph (ds.yParticle.data.dask). Some use of dask.optimize.cull could test that hypothesis.

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?

[pwolfram]

In particular, it appears that the original error is from a different array than xr_var. Are you sure you're pulling out the top level key from the dask dict?

In [214]: ds.yParticle.data.dask
Out[214]: 
{(u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eb',
  0,
  0): (<function dask.array.core.getarray>,
  u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a',
  (slice(0, 31, None), slice(0, 1012000, None))),
 u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a': LazilyIndexedArray(array=LazilyIndexedArray(array=NetCDF4ArrayWrapper(array=<type 'netCDF4._netCDF4.Variable'>
 float64 yParticle(Time, nParticles)
 unlimited dimensions: Time
 current shape = (31, 1012000)
 filling off
 ), key=(slice(None, None, None), slice(None, None, None))), key=(slice(None, None, None), slice(None, None, None)))}

In [215]: x = ds.yParticle.data.dask['fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394
     ...: a']

So I think this is the top level key from the dask dict, but note that this may not be the case

In [219]: ds.yParticle.data.dask.keys()
Out[219]: 
[(u'filename:/yParticle-846a0722e86ecac24903e03f48aa35eb',
  0,
  0),
 u'filename:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a']

In [220]: 

In [220]: x = ds.yParticle.data.dask['In [219]: ds.yParticle.data.dask.keys()
  File "<ipython-input-220-f7fc92c41ba8>", line 1
    x = ds.yParticle.data.dask['In [219]: ds.yParticle.data.dask.keys()
                                                                      ^
SyntaxError: EOL while scanning string literal


In [221]: x = ds.yParticle.data.dask['filename:/yParticle-846a0722e86ecac24903e03f48aa35eb']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-221-94db2c484015> in <module>()
----> 1 x = ds.yParticle.data.dask['filename:/yParticle-846a0722e86ecac24903e03f48aa35eb']

KeyError: 'filename:/yParticle-846a0722e86ecac24903e03f48aa35eb'

[pwolfram]

In [223]: type(ds.yParticle)
Out[223]: xarray.core.dataarray.DataArray

[pwolfram]

@shoyer, can you please clarify:

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?

I don't think this is entirely what you mean:

In [244]: x.compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-244-f4c69c9c1276> in <module>()
----> 1 x.compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'compute'

In [245]: x.array.compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-245-d906770c195f> in <module>()
----> 1 x.array.compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'compute'

In [246]: x.array.array.compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-246-a1bb7b5fa51a> in <module>()
----> 1 x.array.array.compute()

AttributeError: 'NetCDF4ArrayWrapper' object has no attribute 'compute'

In [247]: x.array.array.array.compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-247-975c3b146207> in <module>()
----> 1 x.array.array.array.compute()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getattr__ (netCDF4/_netCDF4.c:36798)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.getncattr (netCDF4/_netCDF4.c:34035)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._get_att (netCDF4/_netCDF4.c:4265)()

AttributeError: NetCDF: Attribute not found

or

In [249]: x.array.array.array.sum().compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-249-63306814fa5b> in <module>()
----> 1 x.array.array.array.sum().compute()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getattr__ (netCDF4/_netCDF4.c:36798)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.getncattr (netCDF4/_netCDF4.c:34035)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._get_att (netCDF4/_netCDF4.c:4265)()

AttributeError: NetCDF: Attribute not found

In [250]: x.array.array.sum().compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-250-77cf68e730e3> in <module>()
----> 1 x.array.array.sum().compute()

AttributeError: 'NetCDF4ArrayWrapper' object has no attribute 'sum'

In [251]: x.array.sum().compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-251-2714bd5c5439> in <module>()
----> 1 x.array.sum().compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'sum'

In [252]: x.sum().compute()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-252-7ce0149d8b09> in <module>()
----> 1 x.sum().compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'sum'

but hopefully this is helpful.

[shoyer]

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?
I don't think this is entirely what you mean:

I mean, e.g.,

chunks = 1e7
da.from_array(xr_var, chunks=chunks, dtype=xr_var.dtype).sum().compute()
da.from_array(xr_var.array, chunks=chunks, dtype=xr_var.array.dtype).sum().compute()
da.from_array(xr_var.array.array, chunks=chunks, dtype=xr_var.array.array.dtype).sum().compute()

I can also make CharToStringArray serialized with dask-distributed on my xarray branch (though as I discussed with @mrocklin today, we will want a slightly different solution later).