Passing file objects to netCDF4.Dataset doesn't work

[rabernat]

I am trying to port some code from scipy.io.netcdf_file to netCDF4.Dataset. I have encountered an issue which is pretty significant for me. netCDF4.Dataset expects a string as its argument and is unable to accept an open file object. The issue can be seen in the following code

import netCDF4
from scipy.io import netcdf_file

fobj = open('MODIS.nc', 'rb')
nc3 = netcdf_file(fobj)
fobj.close()

fobj = open('MODIS.nc', 'rb')
nc4 = netCDF4.Dataset(fobj) # this fails
fobj.close()

The second-to-last line raises

TypeError: expected string or Unicode object, file found

This may seem like an unnecessary feature (why not just pass the filename directly), but the problem is that I have a large archive of bzipped netcdf files on disk. The way I usually read them is

import bz2
bz2_fobj = bz2.BZ2File('MODIS.nc.bz2')
nc3 = netcdf_file(bz2_fobj)

If I can't do this with netCDF4, I will have do design a clumsy workaround involving system commands to manually unzip the files.

I considered tying to add this feature myself, but then I realized that the whole library was written in C. Hopefully you will consider adding support for reading file objects.

[shoyer]

We would also love to have this, but unfortunately, I don't think it's an easy fix (as it would require delving into the netcdf C library). Hopefully @jswhit can elaborate.

[jswhit]

scipy.io_netcdf is a pure python module that reads and writes netcdf-3 formatted files directly. netcdf4-python is a python interface to the netcdf C library, and can handle the netcdf HDF5-based file format. There's no way for the C library to utilize a python file object.

[shoyer]

I doubt there's "no way", but I don't have well defined sense of how difficult it would be (or who actually knows enough to do it). There is, for example, a C side API for working with Python file objects: https://docs.python.org/2/c-api/file.html

[jswhit]

I should have said "impossible without extensive modifications to the HDF5 and netCDF C libs".

I suppose as a workaround we could dump the bytes from the open file object to a temp file, and then pass the name of that temp file to the netCDF C lib.

[shoyer]

I suppose as a workaround we could dump the bytes from the open file object to a temp file, and then pass the name of that temp file to the netCDF C lib.

Indeed, this is the simplest way to solve this problem. But I would say that sort of solution belongs in user code, not this library.

[marqh]

maybe I'm missing something here, but a filehandle has a .name attribute, so could the code to work around this, and then by extension offer the way to fix the issue in the python layer look like:

fobj = open('MODIS.nc', 'rb')
nc4 = netCDF4.Dataset(fobj.name)
fobj.close()

?

It's not especially pretty, but at least it enables expected behaviour to be preserved.

[shoyer]

@marqh that would work for this example, but in general a python file object only needs to adhere the file API -- it need not be an actual file on disk (e.g., it could be a BytesIO object).

[niallrobinson]

Hi everyone - I'd love to see a fix for this. Its coming up quite regularly in "map reduce" world (i.e. Hadoop, Spark, Dask) where we want to be able to pass file objects around and read them quickly, that is without dumping to disk. Is there anything on the horizon that might help out with this?

[dopplershift]

netCDF-C last fall gained support for reading directly from an in-memory buffer that contains the bytes of a netCDF file. It's been on my TODO list to expose this in the Cython wrappers here, but I haven't gotten to it. That's probably your best bet--it's not a file-like object, but at least you wouldn't have to have a file on disk any more.

[niallrobinson]

great - thanks for the update

[jgerardsimcock]

@dopplershift any update on this?

[rabernat]

As the original creator of this issue, I am pleased to see it is still alive. I am still very interested, although more for the reasons described by @niallrobinson. I believe the in-memory buffer solution could solve things. To clarify, would be be able to pass a BytesIO object?

[niallrobinson]

yup - still actively thinking/worrying about this ;)

[dopplershift]

It's still on my todo list, but it hasn't bubbled to the top. I'll try to squeeze it in sooner rather than later (since I don't think it's that hard), but can't make any promises (especially before AMS annual meeting in January).

I don't see BytesIO being supported, since the core functionality would be to read the entire contents of a file into memory and point netCDF at it. BytesIO is about wrapping such a buffer so you can access it like a file. So in my mind it would work like this (borrowing from above):

import bz2
from netCDF4 import Dataset
bz2_fobj = bz2.BZ2File('MODIS.nc4.bz2')
nc4 = Dataset(bz2_fobj.read())

Would that would serve the use cases mentioned here?