BUG: Dask distributed integration tests failing on Travis

[jhamman]

Recent builds on travis are failing for the integration tests for dask distributed (example). Those tests are:

• test_dask_distributed_integration_test[h5netcdf]
• test_dask_distributed_integration_test[netcdf4]

The traceback includes this detail:

_______________ test_dask_distributed_integration_test[netcdf4] ________________
loop = <tornado.platform.epoll.EPollIOLoop object at 0x7fe36dc9e250>
engine = 'netcdf4'
    @pytest.mark.parametrize('engine', ENGINES)
    def test_dask_distributed_integration_test(loop, engine):
        with cluster() as (s, _):
            with distributed.Client(s['address'], loop=loop):
                original = create_test_data()
                with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as filename:
                    original.to_netcdf(filename, engine=engine)
                    with xr.open_dataset(filename, chunks=3, engine=engine) as restored:
                        assert isinstance(restored.var1.data, da.Array)
>                       computed = restored.compute()
xarray/tests/test_distributed.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
xarray/core/dataset.py:487: in compute
    return new.load()
xarray/core/dataset.py:464: in load
    evaluated_data = da.compute(*lazy_data.values())
../../../miniconda/envs/test_env/lib/python2.7/site-packages/dask/base.py:206: in compute
    results = get(dsk, keys, **kwargs)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1923: in get
    results = self.gather(packed, asynchronous=asynchronous)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1368: in gather
    asynchronous=asynchronous)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:540: in sync
    return sync(self.loop, func, *args, **kwargs)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/utils.py:239: in sync
    six.reraise(*error[0])
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/utils.py:227: in f
    result[0] = yield make_coro()
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/gen.py:1055: in run
    value = future.result()
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/concurrent.py:238: in result
    raise_exc_info(self._exc_info)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/tornado/gen.py:1063: in run
    yielded = self.gen.throw(*exc_info)
../../../miniconda/envs/test_env/lib/python2.7/site-packages/distributed/client.py:1246: in _gather
    traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   c = a[b]
E   TypeError: string indices must be integers

Distributed v.1.18.1 was released 5 days ago so there must have been a breaking change that has been passed down to us.

cc @shoyer, @mrocklin

[shoyer]

Interestingly they only seems to be failing on Python 2.7.

[mrocklin]

I was not able to reproduce immediately on my local environment. I'll try to reproduce with the travis-ci environment next time.

…

On Thu, Aug 31, 2017 at 1:21 PM, Stephan Hoyer ***@***.***> wrote: Interestingly they only seems to be failing on Python 2.7. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1540 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszGSLrI9B_pCt-jDD1w8PGPYiI0opks5sdut8gaJpZM4PIVDJ> .

[spencerahill]

One bit that may be helpful: this is arising in aospy's tests, e.g. here. But, they're only occurring in our 'py27-min' environment (and are doing so with 100% consistency); never in our full 'py27' environment.

environment-py27-min.yml:

name: test_env
channels:
  - conda-forge
dependencies:
  - python=2.7
  - scipy
  - netCDF4
  - xarray
  - dask
  - distributed
  - pip:
    - coveralls
    - pytest-cov

environment-py27.yml

name: test_env
channels:
  - conda-forge
dependencies:
  - python=2.7
  - scipy
  - netCDF4
  - xarray
  - dask
  - distributed
  - pytest
  - future
  - matplotlib
  - ipython
  - pip:
    - coveralls
    - pytest-cov
    - pytest-catchlog
    - runipy

[mrocklin]

I've identified the problem and will have a fix soon. There was another issue with this release so I'll probably issue a micro-release shortly.

…

On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> wrote: One bit that may be helpful: this is arising in aospy's tests, e.g. here <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. But, they're only occurring in our 'py27-min' environment (and are doing so with 100% consistency); never in our full 'py27' environment. environment-py27-min.yml <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> : name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pip: - coveralls - pytest-cov environment-py27.yml <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pytest - future - matplotlib - ipython - pip: - coveralls - pytest-cov - pytest-catchlog - runipy — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1540 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> .

[mrocklin]

shortly -> within a few days

…

On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: I've identified the problem and will have a fix soon. There was another issue with this release so I'll probably issue a micro-release shortly. On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> wrote: > One bit that may be helpful: this is arising in aospy's tests, e.g. here > <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. > But, they're only occurring in our 'py27-min' environment (and are doing so > with 100% consistency); never in our full 'py27' environment. > > environment-py27-min.yml > <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> > : > > name: test_env > channels: > - conda-forge > dependencies: > - python=2.7 > - scipy > - netCDF4 > - xarray > - dask > - distributed > - pip: > - coveralls > - pytest-cov > > environment-py27.yml > <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> > > name: test_env > channels: > - conda-forge > dependencies: > - python=2.7 > - scipy > - netCDF4 > - xarray > - dask > - distributed > - pytest > - future > - matplotlib > - ipython > - pip: > - coveralls > - pytest-cov > - pytest-catchlog > - runipy > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1540 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> > . >

[mrocklin]

This should be resolved in dask/distributed#1370

[mrocklin]

To properly debug this I had to make an async test (most dask.distributed testing does this, search for gen_cluster if curious). This was harder than I would have liked with xarray. We might consider switching some things around to be agnostic to synchronous/asynchronous APIs in the near to moderate future (admittedly, most of this work is probably on me).

…

On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: shortly -> within a few days On Thu, Aug 31, 2017 at 3:14 PM, Matthew Rocklin ***@***.***> wrote: > I've identified the problem and will have a fix soon. There was another > issue with this release so I'll probably issue a micro-release shortly. > > On Thu, Aug 31, 2017 at 2:43 PM, Spencer Hill ***@***.***> > wrote: > >> One bit that may be helpful: this is arising in aospy's tests, e.g. here >> <https://travis-ci.org/spencerahill/aospy/builds/270528201?utm_source=github_status&utm_medium=notification>. >> But, they're only occurring in our 'py27-min' environment (and are doing so >> with 100% consistency); never in our full 'py27' environment. >> >> environment-py27-min.yml >> <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27-min.yml> >> : >> >> name: test_env >> channels: >> - conda-forge >> dependencies: >> - python=2.7 >> - scipy >> - netCDF4 >> - xarray >> - dask >> - distributed >> - pip: >> - coveralls >> - pytest-cov >> >> environment-py27.yml >> <https://github.com/spencerahill/aospy/blob/develop/ci/environment-py27.yml> >> >> name: test_env >> channels: >> - conda-forge >> dependencies: >> - python=2.7 >> - scipy >> - netCDF4 >> - xarray >> - dask >> - distributed >> - pytest >> - future >> - matplotlib >> - ipython >> - pip: >> - coveralls >> - pytest-cov >> - pytest-catchlog >> - runipy >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#1540 (comment)>, >> or mute the thread >> <https://github.com/notifications/unsubscribe-auth/AASszAePI7b7ThHzz5iQA7k9Ku9roRdpks5sdv7NgaJpZM4PIVDJ> >> . >> > >