FileNotFoundError when using FsspecStore.from_url with a sync filesystem

[maxrjones]

Zarr version

'3.0.3.dev9+g90cc08d4.d20250206'

Numcodecs version

'0.15.0'

Python Version

3.13

Operating System

Mac

Installation

Following https://zarr.readthedocs.io/en/latest/developers/contributing.html#creating-a-development-environment

Description

Writing an array to FsspecStore containing a AsyncFileSystemWrapper object does not currently work.

Steps to reproduce

import zarr
from zarr.storage import FsspecStore
from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
import tempfile
import numpy as np

with tempfile.TemporaryDirectory() as tmp_path:
    store = FsspecStore.from_url(f"local://{tmp_path}")
    assert isinstance(store.fs, AsyncFileSystemWrapper)
    assert store.fs.async_impl
    z = zarr.create_array(
        store=store,
        shape=(100, 100),
        chunks=(10, 10),
        dtype="f4"
    )
    # write to the array
    z[:, :] = np.random.random((100, 100))

Additional output

I added some print statements to check that the mode is correct for all the fsspec opener functions, which is consistently 'wb'. Here's the full traceback excluding those print statements:

  File "/Users/max/Documents/Code/zarr-developers/zarr-python/.vscode/test-async-wrapper.py", line 18, in <module>
    z[:, :] = np.random.random((100, 100))
    ~^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 2523, in __setitem__
    self.set_orthogonal_selection(pure_selection, value, fields=fields)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 2979, in set_orthogonal_selection
    return sync(
        self._async_array._set_selection(indexer, value, fields=fields, prototype=prototype)
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/sync.py", line 142, in sync
    raise return_result
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/sync.py", line 98, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/array.py", line 1413, in _set_selection
    await self.codec_pipeline.write(
    ...<11 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 468, in write
    await concurrent_map(
    ...<6 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 68, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 66, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 418, in write_batch
    await concurrent_map(
    ...<8 lines>...
    )
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 68, in concurrent_map
    return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/common.py", line 66, in run
    return await func(*item)
           ^^^^^^^^^^^^^^^^^
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/core/codec_pipeline.py", line 416, in _write_key
    await byte_setter.set(chunk_bytes)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/storage/_common.py", line 144, in set
    await self.store.set(self.path, value)
  File "/Users/max/Documents/Code/zarr-developers/zarr-python/src/zarr/storage/_fsspec.py", line 276, in set
    await self.fs._pipe_file(path, value.to_bytes())
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/asyn_wrapper.py", line 27, in wrapper
    return await asyncio.to_thread(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/miniforge3/envs/zarr-python-tests/lib/python3.13/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/miniforge3/envs/zarr-python-tests/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/spec.py", line 787, in pipe_file
    with self.open(path, "wb", **kwargs) as f:
         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        path,
    ...<4 lines>...
        **kwargs,
    )
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 200, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 364, in __init__
    self._open()
    ~~~~~~~~~~^^
  File "/Users/max/Library/Application Support/hatch/env/virtual/zarr/M9h0Qn3b/test.py3.13-2.1-optional/lib/python3.13/site-packages/fsspec/implementations/local.py", line 369, in _open
    self.f = open(self.path, mode=self.mode)
             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/70/hc_nynms54d8lp67z4rsfctc0000gp/T/tmpjmu3h688/c/0/2'

[martindurant]

The parent folder didn't get created, maybe?

[maxrjones]

The parent folder didn't get created, maybe?

Yeah I cannot say I understand it, but fwiw the metadata file gets created properly so my best guess is that there's an issue with the "c" chunks subdirectory.

[martindurant]

Try passing auto_mkdir=True to the filesystem, or explicitly create the directories first.

[will-moore]

I've come across the same error, originally coming from dask.array to_zarr() at dask/dask#11691 but now with updated zarr and removed the dask call to give this:

zarr==v3.03
fsspec==2024.12.0

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False)

z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

Similar stack trace as above

  File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 17, in <module>
    z = zarr.create(
        ^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 708, in create
    sync(
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 163, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1044, in create
    return await AsyncArray._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 611, in _create
    result = await cls._create_v3(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 754, in _create_v3
    await array._save_metadata(metadata, ensure_parents=True)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 1369, in _save_metadata
    await gather(*awaitables)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/abc/store.py", line 487, in set_or_delete
    await byte_setter.set(value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_common.py", line 144, in set
    await self.store.set(self.path, value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_fsspec.py", line 277, in set
    await self.fs._pipe_file(path, value.to_bytes())
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/asyn_wrapper.py", line 27, in wrapper
    return await asyncio.to_thread(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 787, in pipe_file
    with self.open(path, "wb", **kwargs) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 200, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 364, in __init__
    self._open()
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 369, in _open
    self.f = open(self.path, mode=self.mode)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.zarr/zarr.json'

[will-moore]

Just an update... Still seeing this with:

fsspec==2025.3.0
zarr==3.0.5

I don't see that I can use auto_mkdir=True in this example:

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False)
z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

[martindurant]

The following does work:

import zarr
store = zarr.storage.FsspecStore.from_url("test_dask_to_zarr.zarr", read_only=False, storage_options={"auto_mkdir": True})
z = zarr.create(
    shape=(256, 256),
    chunks=(64, 64),
    dtype="int8",
    store=store
)

Note that the fsspec config allows you to define that auto_mkdir should always be True for local FS, if you want.

[will-moore]

@martindurant Thanks for that. I've closed my original issue at dask/dask#11691 since it works with storage_options={"auto_mkdir": True}, but I won't close this issue just yet as it's not "my" issue originally. cc @maxrjones