MemoryError when sequentially loading big data sets

[rphlypo]

Although each of the unzipped data sets fits RAM size, calling a sequence of functions and deleting each object after having written it to disk, raises a MemoryError of which the traceback is given below. My data (gzipped Nifti1 format with extension .nii.gz) contains about 1GB of gzipped data for each data set and my machine has a 16GB of RAM, still it crashes during manipulation of each third data set, with the exact same message (error in the same line of code, same traceback).

Is this due to the data handling in gzip, or due to the asanyarray constructor that provides too much overhead?

Thanks in advance for any return on this issue.

~/.local/lib/python2.7/site-packages/nibabel/spatialimages.pyc in get_data(self)
339
340 def get_data(self):
--> 341 return np.asanyarray(self._data)
342
343 @Property

/usr/lib/python2.7/dist-packages/numpy/core/numeric.pyc in asanyarray(a, dtype, order)
285
286 """
--> 287 return array(a, dtype, copy=False, order=order, subok=True)
288
289 def ascontiguousarray(a, dtype=None):

~/.local/lib/python2.7/site-packages/nibabel/arrayproxy.pyc in array(self)
53 ''' Cached read of data from file '''
54 if self._data is None:
---> 55 self._data = self._read_data()
56 return self._data
57

~/.local/lib/python2.7/site-packages/nibabel/arrayproxy.pyc in _read_data(self)
58 def _read_data(self):
59 fileobj = allopen(self.file_like)
---> 60 data = self.header.data_from_fileobj(fileobj)
61 if isinstance(self.file_like, basestring): # filename
62 fileobj.close()

~/.local/lib/python2.7/site-packages/nibabel/analyze.pyc in data_from_fileobj(self, fileobj)
484 '''
485 # read unscaled data

--> 486 data = self.raw_data_from_fileobj(fileobj)
487 # get scalings from header. Value of None means not present in header

488 slope, inter = self.get_slope_inter()

~/.local/lib/python2.7/site-packages/nibabel/analyze.pyc in raw_data_from_fileobj(self, fileobj)
456 shape = self.get_data_shape()
457 offset = self.get_data_offset()
--> 458 return array_from_file(shape, dtype, fileobj, offset)
459
460 def data_from_fileobj(self, fileobj):

~/.local/lib/python2.7/site-packages/nibabel/volumeutils.pyc in array_from_file(shape, in_dtype, infile, offset, order)
482 if datasize == 0:
483 return np.array([])
--> 484 data_str = infile.read(datasize)
485 if len(data_str) != datasize:
486 if hasattr(infile, 'name'):

/usr/lib/python2.7/gzip.pyc in read(self, size)
254 try:
255 while size > self.extrasize:
--> 256 self._read(readsize)
257 readsize = min(self.max_read_chunk, readsize * 2)
258 except EOFError:

/usr/lib/python2.7/gzip.pyc in _read(self, size)
306
307 uncompress = self.decompress.decompress(buf)
--> 308 self._add_read_data( uncompress )
309
310 if self.decompress.unused_data != "":

/usr/lib/python2.7/gzip.pyc in _add_read_data(self, data)
324 self.crc = zlib.crc32(data, self.crc) & 0xffffffffL
325 offset = self.offset - self.extrastart
--> 326 self.extrabuf = self.extrabuf[offset:] + data
327 self.extrasize = self.extrasize + len(data)
328 self.extrastart = self.offset

MemoryError:

[matthew-brett]

Sorry to be slow to get back to you.

When you do 'get_data' on an image, nibabel will load the image array into memory, return it, and also keep a reference to the array in the image. So, if you want the image array data to get deleted from memory, you need to delete all references to the array - including the image object.

You can avoid keeping a reference to the loaded array in the lastest nibabel master by using np.asarray(img.dataobj) which returns the array, but does not keep a reference to the array

[GaelVaroquaux]

Hi @matthew-brett : I am looking at this issue with @rphlypo (who works in my lab).

Your answer is interesting, but it is suboptimal if the data is in cache. Right now we are doing something like:

if niimg._data_cache is None:
   data = np.asarray(img.dataobj)
else:

[GaelVaroquaux]

Hi @matthew-brett : I am looking at this issue with @rphlypo (who works in my lab).

Your answer is interesting, but it is suboptimal if the data is in cache. Right now we are doing something like:

if niimg._data_cache is None:
   data = np.asarray(img.dataobj)
else:
   data = niimg.det_data()

I don't really like this, because we are addressing a private attribute. I feel that something like:

  data = niimg.get_data(cache=False)

would be a useful (and easy) addition to nibabel.

I am willing to contribute it, if you agree with it.

[GaelVaroquaux]

Previous comment modified, as I had sent it too early. Please check it on the web, rather than in your inboxes.

[GaelVaroquaux]

And, by the way, the discussion that we are having is useful, but doesn't address @rphlypo 's problem, which is that nibabel blows the memory while decompressing the data from the disk. I'll open an ENH issue with an analaysis of the problem and a proposition of a way to improve the situation.

[matthew-brett]

Thanks for the discussion. I'm in Cuba at the moment - will do my best to keep up.

The cache=False idea is not a bad one - meaning - return from cache if possible, otherwise read from disk, but don't store in cache. Sounds good to me.

[matthew-brett]

Does this PR help?

#211

[GaelVaroquaux]

PR #211 wouldn't help because it is concerned with partial loading, whereas what we want to do if full loading. However, it wouldn't help even for partial loading and would reproduce the exact same problem if people are trying to address the end of the file.

The reason is that gzip isn't a seekable compression format. As a result, the "seek" method of gzip.GzipFile uses the "read" method, and that calls the infamous line (line 326 of gzip.py in my Python 2.7 install):

        self.extrabuf = self.extrabuf[offset:] + data

in _add_read_data.

This would be fixable by patching the standard lib.

[matthew-brett]

Can I close this one then? Or can you think of another fix?

[matthew-brett]

Gael - can I close this issue?

[matthew-brett]

Gael - looking at this again, I agree we need something on this line.

I was thinking of exposing an is_cached attribute, so you can use your code as is:

if not niimg.is_cached:
   data = np.asarray(img.dataobj)
else:
   data = niimg.det_data()

This seems nice and explicit, and avoids the confusion of the case where `img.get_data(cache=False)`` gets run on an image which is already cached.

We're coming up to a release, so please let me know ASAP.

Comments anyone?

[matthew-brett]

Or maybe img.in_memory returning True for an image constructed against an array as well as a cached array from a proxy image after get_data()?

[effigies]

@GaelVaroquaux If this is still an issue you're hacking around that would be best addressed in nibabel, please reopen.