Skip to content

ENH: fast reads on large gzip files #210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 13, 2014
Merged

ENH: fast reads on large gzip files #210

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
46122c4
ENH: fast reads on large gzip files
GaelVaroquaux Nov 14, 2013
dee72f8
BUG: fix previous commit
GaelVaroquaux Nov 18, 2013
69edcc9
RF: put gzip chunk constant into module global
matthew-brett Jul 29, 2014
4c3572b
Merge pull request #1 from matthew-brett/module-global-gzip
GaelVaroquaux Oct 13, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion nibabel/openers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@
import gzip
import bz2

# The largest memory chunk that gzip can use for reads
GZIP_MAX_READ_CHUNK = 100 * 1024 * 1024 # 100Mb


def _gzip_open(fileish, *args, **kwargs):
# open gzip files with faster reads on large files using larger chunks
# See https://github.com/nipy/nibabel/pull/210 for discussion
gzip_file = gzip.open(fileish, *args, **kwargs)
gzip_file.max_read_chunk = GZIP_MAX_READ_CHUNK
return gzip_file


class Opener(object):
""" Class to accept, maybe open, and context-manage file-likes / filenames
Expand All @@ -32,7 +43,7 @@ class Opener(object):
passed to opening method when `fileish` is str. Change of defaults as
for \*args
"""
gz_def = (gzip.open, ('mode', 'compresslevel'))
gz_def = (_gzip_open, ('mode', 'compresslevel'))
bz2_def = (bz2.BZ2File, ('mode', 'buffering', 'compresslevel'))
compress_ext_map = {
'.gz': gz_def,
Expand Down