Installation on Windows

[Trigonometry]

At first I want to say thanks for this great improvement to directly fast-read and extract content-parts in huge gzip-compressed files. igzip.IndexedGzipFile() should be the new standard instead of gzip.open() Installation and testing on Linux works like a charm, but ...
I am mainly writing and testing python-programs on JetBrains pyCharm on a Windows platform but this specific python package fails either by directly adding the package to the project interpreter in pyCharm, or manually installing it on command line:

python -m pip install indexed-gzip
Collecting indexed-gzip
Using cached indexed_gzip-0.3.1.tar.gz
Building wheels for collected packages: indexed-gzip
Running setup.py bdist_wheel for indexed-gzip ... error
Complete output from command C:\Python27\python.exe -u -c "import setuptools, tokenize;file='%APPDATA%\local\temp\pip-build-d9mi7z\indexed-gzip\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replac
e('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d %APPDATA%\local\temp\tmpqdae
xbpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'indexed_gzip' extension
creating build
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tcindexed_gzip.c /Fobuild\temp.win32-2.7\Release\indexed_gzip.obj -Wno-unused-function
cl : Command line error D8021 : invalid numeric argument '/Wno-unused-function'
error: command '%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe' failed with exit status 2
...

Extracting the latest indexed_gzip-master.zip from GitHub and commenting out the -Wno-unused-function parameter for directly pip installing from folder fails during compiling zran.c

python -m pip install -e ./indexed_gzip
Obtaining file:///C:/Temp/indexed_gzip
Installing collected packages: indexed-gzip
Running setup.py develop for indexed-gzip
Complete output from command C:\Python27\python.exe -c "import setuptools, tokenize;file='C:\Temp\indexed_gzip\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, _file
_, 'exec'))" develop --no-deps:
running develop
running egg_info
writing indexed_gzip.egg-info\PKG-INFO
writing top-level names to indexed_gzip.egg-info\top_level.txt
writing dependency_links to indexed_gzip.egg-info\dependency_links.txt
reading manifest file 'indexed_gzip.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'bench_plot.py'
warning: no previously-included files found matching 'benchmark_indexed_gzip.py'
writing manifest file 'indexed_gzip.egg-info\SOURCES.txt'
running build_ext
building 'indexed_gzip' extension
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC \Tcindexed_gzip.c /Fobuild\temp.win32-2.7\Release\indexed_gzip.obj
indexed_gzip.c
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdint.h(268) : warning C4005: 'SIZE_MAX' : macro redefinition
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\limits.h(84) : see previous definition of 'SIZE_MAX'
indexed_gzip.c(1702) : warning C4244: 'function' : conversion from 'long' to 'uint16_t', possible loss of data
indexed_gzip.c(3866) : warning C4244: 'function' : conversion from 'int64_t' to 'Py_ssize_t', possible loss of data
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tczran.c /Fobuild\temp.win32-2.7\Release\zran.obj
zran.c
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdint.h(268) : warning C4005: 'SIZE_MAX' : macro redefinition
%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\limits.h(84) : see previous definition of 'SIZE_MAX'
zran.c(468) : warning C4013: 'fcntl' undefined; assuming extern returning int
zran.c(468) : error C2065: 'F_GETFL' : undeclared identifier
zran.c(468) : error C2065: 'O_ACCMODE' : undeclared identifier
zran.c(474) : warning C4013: 'fseeko' undefined; assuming extern returning int
zran.c(477) : warning C4013: 'ftello' undefined; assuming extern returning int
zran.c(497) : warning C4244: '=' : conversion from 'int64_t' to 'size_t', possible loss of data
zran.c(501) : warning C4013: 'round' undefined; assuming extern returning int
zran.c(540) : error C2143: syntax error : missing ';' before 'type'
zran.c(543) : error C2065: 'new_list' : undeclared identifier
zran.c(543) : warning C4047: '==' : 'int' differs in levels of indirection from 'void *'
zran.c(547) : error C2065: 'new_list' : undeclared identifier
zran.c(547) : warning C4047: '=' : 'zran_point_t *' differs in levels of indirection from 'int'
zran.c(559) : error C2143: syntax error : missing ';' before 'type'
zran.c(561) : error C2065: 'new_list' : undeclared identifier
zran.c(561) : warning C4047: '=' : 'int' differs in levels of indirection from 'void *'
zran.c(563) : error C2065: 'new_list' : undeclared identifier
zran.c(563) : warning C4047: '==' : 'int' differs in levels of indirection from 'void *'
zran.c(567) : error C2065: 'new_list' : undeclared identifier
zran.c(567) : warning C4047: '=' : 'zran_point_t *' differs in levels of indirection from 'int'
zran.c(630) : warning C4244: '=' : conversion from 'uint64_t' to 'uint32_t', possible loss of data
zran.c(870) : warning C4244: '=' : conversion from 'double' to 'uint64_t', possible loss of data
zran.c(871) : warning C4244: '=' : conversion from 'double' to 'uint64_t', possible loss of data
zran.c(924) : error C2143: syntax error : missing ';' before 'type'
zran.c(925) : error C2275: 'zran_point_t' : illegal use of this type as an expression
c:\temp\indexed_gzip\zran.h(22) : see declaration of 'zran_point_t'
zran.c(925) : error C2065: 'next' : undeclared identifier
zran.c(943) : error C2065: 'point_data' : undeclared identifier
zran.c(943) : warning C4047: '=' : 'int' differs in levels of indirection from 'void *'
zran.c(946) : error C2065: 'point_data' : undeclared identifier
zran.c(946) : warning C4047: '=' : 'int' differs in levels of indirection from 'void *'
zran.c(947) : error C2065: 'point_data' : undeclared identifier
zran.c(947) : warning C4047: '==' : 'int' differs in levels of indirection from 'void *'
zran.c(951) : error C2065: 'next' : undeclared identifier
zran.c(951) : warning C4047: '=' : 'int' differs in levels of indirection from 'zran_point_t *'
zran.c(952) : error C2065: 'next' : undeclared identifier
zran.c(952) : error C2223: left of '->bits' must point to struct/union
zran.c(953) : error C2065: 'next' : undeclared identifier
zran.c(953) : error C2223: left of '->cmp_offset' must point to struct/union
zran.c(954) : error C2065: 'next' : undeclared identifier
zran.c(954) : error C2223: left of '->uncmp_offset' must point to struct/union
zran.c(955) : error C2065: 'next' : undeclared identifier
zran.c(955) : error C2223: left of '->data' must point to struct/union
zran.c(955) : error C2065: 'point_data' : undeclared identifier
zran.c(969) : error C2065: 'point_data' : undeclared identifier
zran.c(969) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 1
zran.c(977) : error C2065: 'point_data' : undeclared identifier
zran.c(977) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 1
zran.c(981) : error C2065: 'point_data' : undeclared identifier
zran.c(981) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 1
zran.c(1001) : error C2065: 'point_data' : undeclared identifier
zran.c(1001) : warning C4047: '!=' : 'int' differs in levels of indirection from 'void *'
zran.c(1002) : error C2065: 'point_data' : undeclared identifier
zran.c(1002) : warning C4022: 'free' : pointer mismatch for actual parameter 1
zran.c(1038) : warning C4244: '=' : conversion from 'uint64_t' to 'off_t', possible loss of data
zran.c(1599) : warning C4244: '=' : conversion from 'uint64_t' to 'size_t', possible loss of data
zran.c(2006) : warning C4244: '+=' : conversion from 'uint64_t' to 'off_t', possible loss of data
zran.c(2025) : warning C4244: '=' : conversion from 'uint64_t' to 'off_t', possible loss of data
zran.c(2058) : warning C4244: 'return' : conversion from 'uint64_t' to 'long', possible loss of data
zran.c(2155) : warning C4244: 'function' : conversion from 'uint64_t' to 'size_t', possible loss of data
zran.c(2165) : warning C4244: '=' : conversion from 'uint64_t' to 'off_t', possible loss of data
zran.c(2166) : warning C4244: '=' : conversion from 'uint64_t' to 'off_t', possible loss of data
zran.c(2214) : warning C4244: 'function' : conversion from 'uint64_t' to 'uint32_t', possible loss of data
zran.c(2294) : warning C4244: 'function' : conversion from 'uint64_t' to 'uint32_t', possible loss of data
zran.c(2295) : error C2036: 'void *' : unknown size
error: command '%APPDATA%\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe' failed with exit status 2

Environment:
°°°°°°°°°°°°°°°°
pyCharm 2016.2.3
python 2.7.8
numpy 1.12.1 (above requirements)
cython 0.25.2 (above requirements)
Visual C++ 9.0 Compiler for Python 2.7
Win7 Ultimate 32bit (don't laugh)

Hope you have an idea how to solve this issue to get indexed-gzip installed and running on Windows too since there are sadly no pre-compiled windows binaries provided for this specific package on Christoph Golhkes' "Unofficial Windows Binaries for Python Extension Packages" website (yet). Thanks for the great work and keep it up .. :-)

[zrajna]

This seems to be a MS Visual C compiler specific issue.

• It does not have a /Wno-unused-function flag.
• In Windows the determination of read-only file state is a bit different, it needs to be implemented separately (zran.c:468). See discussion here for example: http://stackoverflow.com/questions/4636875/on-windows-mingw-what-is-the-equivalent-of-fcntlfd-f-getfl-o-accmode

I would try it with GCC on Windows (e.g. cygwin), but I don't have Windows...

[pauldmccarthy]

Huh, Windows support hadn't even crossed my mind! Unfortunately I don't have access to a Windows machine either, and know absolutely nothing about developing on/for Windows.

@Trigonometry, would you be able to try the approach linked to by @zrajna ?

[Trigonometry]

Thank you both for the quick answers. With a little digging and the links you provided I could get rid of most of the Compiling Errors and ignored the possible loss of data during type conversation warnings. I attached the rewriting process in a plain txt file because inserting and reformating it here would be too much (for me and for you).

rewrite_win_edit.txt

I still have 2 issues during linking to the two external python packages, which I was unable to solve, but maybe you as the developer have a more insight view of these function calls.

unresolved external symbols:
_mmap referenced in function _pyx_pf_5tests_10ctest_zran_10ReadBuffer___cinit
_munmap referenced in function _pyx_pf_5tests_10ctest_zran_10ReadBuffer_4__dealloc

Thanks for any help. I come closer, I can already can taste the win-build .. :-D Have a nice weekend .. :)

[Trigonometry]

The last problem seems to be a bit difficult. The reason for the unresolved external symbols is an outdated win32 precompiled zlib.lib file from zlib v.1.2.3 (year 2005) which didn't contain the functions mmap and munmap back in the days. I was able to reproduce similar unresolved externals errors when I re-attempt at home on a 64bit Windows:

unresolved external symbol inflateSetDictionary referenced in function _zran_init_zlib_inflate
unresolved external symbol inflatePrime referenced in function zran_init_zlib_inflate
unresolved external symbol inflateInit2 referenced in function _zran_init_zlib_inflate
unresolved external symbol inflateEnd referenced in function _zran_find_next_stream
unresolved external symbol inflate referenced in function _zran_inflate

I guess that leaves it to self-compile the latest sources for zlib (1.2.11 I think) before getting any further on the indexed_gzip for windows users .. :-)

[pauldmccarthy]

The mmap/munmap calls are actually only used in the test cases, so you might have some success just by disabling their compilation.

As for your later issues (the errors on your 64 bit machine), that looks like a linking issue - those functions should all be present if zlib is present.

But, more generally, I'm afraid I can't make the changes in your proposed patch, simply because I don't have a windows machine to test on (and https://travis-ci.org doesn't have support for windows either. I would really need a PR which will pass all tests on all platforms...

[Trigonometry]

No problem at all .. :-) I just got it working. As soon as the code was modified, I was at least able to cythonize the extensions and generate the .egg. With the generated indexed_gzip.c it was only 2 commands and a few steps aways for a system integration:

python -m pip install ./indexed_gzip-master
Processing ..\indexed_gzip-master
Installing collected packages: indexed-gzip
Running setup.py install for indexed-gzip ... done
Successfully installed indexed-gzip-0.3.1

cd indexed_gzip-master/.eggs
easy_install pytest_runner-2.11.1-py2.7.egg
Processing pytest_runner-2.11.1-py2.7.egg
Copying pytest_runner-2.11.1-py2.7.egg to c:\python27\lib\site-packages
pytest-runner 2.11.1 is already the active version in easy-install.pth
Installed c:\python27\lib\site-packages\pytest_runner-2.11.1-py2.7.egg
Processing dependencies for pytest-runner==2.11.1
Finished processing dependencies for pytest-runner==2.11.1

As import indexed_gzip on python console threw a ImportError: DLL load failed I could identify the missing libraries using:

python
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL('indexed_gzip.pyd')
zlibd.dll was missing in /Windows/System32/ which I copied over from the compilation of the latest zlib source v1.2.11 and another (ucrtbase.dll if I recall correctly). After that the import worked and testing too:

import indexed_gzip as igzip

myfile = igzip.IndexedGzipFile(filename='file.gz')
some_offset_into_uncompressed_data = 234195000

myfile.seek(some_offset_into_uncompressed_data)
data = myfile.read(104857)
print data
...
e germline GRCh38 NC_000011.10 11 18288021 18288021 A G 11p15.1 criteria provided, single submitter 1N - 2313475 single nucleotide variant
...

I will try to reproduce the steps on 32bit when I am back at work .. :-) Thanks for giving me some good hints on the way for the solution and enjoy the weekend (not as I do .. :-P).

[pauldmccarthy]

Nice work ... it would be great if you could share the changes that you have made, so that I (or somebody else) might be able to add Windows support in the future.

[Trigonometry]

Indexed-gzip on 64bit Windows

Modifications before compilation

setup.py
°°°°°°°°°°°
// to avoid compiling tests, which fail
50: have_numpy = False
56: #try: import numpy as np
57: #except: have_numpy = False

// enable large file offsets on Windows (IMPORTANT)
78: extra_compile_args=['-Wall', '-D_FILE_OFFSET_BITS=64', '-pedantic', '-Wno-unused-function'])

zran.c
°°°°°°°
// not known in Windows
468: /* if ((fcntl(fileno(fd), F_GETFL) & O_ACCMODE) != O_RDONLY)**
469: goto fail; */

indexed_gzip.pyx
°°°°°°°°°°°°°°°°°°°°°
// Allow to receive correct data type/value of index->uncmp_seek_offset from zran_tell()
// to calculate extracted filesize without decompressing the file to disk
206: cdef unsigned long r = zran.zran_tell(&self.index)
207: return r

Compiler

MinGW-Compiler
°°°°°°°°°°°°°°°°°°°°°°
python -m pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
Downloading https://pypi.anaconda.org/carlkl/simple/mingwpy/0.1.0b3/mingwpy-0.1.0b3-cp27-none-win32.whl (74.0MB)
100% |################################| 74.1MB 9.5kB/s
Installing collected packages: mingwpy
Successfully installed mingwpy-0.1.0b3

// Make sure C:\Python27\Scripts is in PATH to correctly find gcc

Compilation / Installation

python -m pip install --global-option build_ext --global-option --compiler=mingw32 ./indexed_gzip-master
Processing d:\igzip\indexed_gzip-master
Skipping bdist_wheel for indexed-gzip, due to binaries being disabled for it.
Installing collected packages: indexed-gzip
Running setup.py install for indexed-gzip ... done
Successfully installed indexed-gzip-0.3.3

Generate 32/64bit Wheel(s) for Installation on other Windows-Clients (Optional)
°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
python -m pip wheel --global-option build_ext --global-option --compiler=mingw32 ./indexed_gzip-master
Processing d:\igzip\indexed_gzip-master
Building wheels for collected packages: indexed-gzip
Running setup.py bdist_wheel for indexed-gzip ... done
Stored in directory: d:\igzip
Successfully built indexed-gzip

Wheel-Installation on other Clients (Optional)
°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
python -m pip install indexed_gzip-0.3.3-cp27-cp27m-win_amd64.whl
Processing d:\igzip\indexed_gzip-0.3.3-cp27-cp27m-win_amd64.whl
Installing collected packages: indexed-gzip
Successfully installed indexed-gzip-0.3.3

Testing

python
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import indexed_gzip
>>>

Environment

pyCharm 2017.1.2
python 2.7.13
pip 9.0.1
mingwpy 0.1.0b3
numpy 1.12.1 (above requirements)
cython 0.25.2 (above requirements)
Win10 Home 64bit

To-Do's
°°°°°°°°°°
1. get compilation of optional tests working
2. rewriting unix fcntl() for windows

[Trigonometry]

Sorry for the downtime but I was busy installing 2x 64bit Systems on a more powerful PC to be able to index large gz-Files by myself on Linux and Windows since we stumbled across 2 weird behaviors on 32bit, which we thought would be fixed on 64bit but that was not the case. I just updated the latest working attempt on a 64bit-System that should work on any 64bit Windows now in the previous comment. (32bit should work too, except the return of the decompressed filesize maybe).

The first error I produced myself during the attempt to rewrite unix check if the file is openend in read-only mode if ((fcntl(fileno(fd), F_GETFL) & O_ACCMODE) != O_RDONLY) with windows api-call in zran.c:

#include <winsock2.h>
int iResult;
u_long off = 1;
iResult = ioctlsocket(fileno(fd), FIONBIO, &off);
if (iResult != NO_ERROR)
    goto fail;

Since igzip.IndexedGzipFile(filename=filepath) only sporadic could open .gz files we tracked the mistake down using zran_log("ioctl_socket_result=%i, %i\n", iResult, WSAGetLastError()); to this wrong call. At this time since we can't use fcntl() from Unix and ioctlsocket() on a file descriptor is not working either, we don't check if the file is openend in read-only mode.

The second error was a really time and coffee consuming error during a wrong return of an already existing value for the decompressed filesize index->uncmp_seek_offset. Since you already build that value by extending it by each successfull read-block we thought it would be a great improvement to know how big the extracted content would be for seek and the ability to read last x extracted bytes. While the modern gz-format does not store that value together with the checksum explicit in the last 8bytes in the compressed format anymore you can now attempt to seek behind the decompressed size and return the last valid read-position:

import indexed_gzip as igzip

filepath = 'D:/gene2refseq.gz'
f = igzip.IndexedGzipFile(filename=filepath)

pot = 2**33          # usually ~8GB should be enough
try:
        f.seek(pot)
        print "File bigger than seek position."
except Exception:
        print f.tell()

Compr: 549.948.956 <type 'long'>
Seek: 8.589.934.592 Bytes <type 'long'>
Decompr: 3.346.495.618 Bytes <type 'long'>

The fix for the wrong return value (returned signed int before) is:
a) the compiler flag -D_FILE_OFFSET_BITS=64
b) a variable cast in the tell() function in indexed_gzip.pyx

def tell(self):
    """Returns the current seek offset into the uncompressed data stream.
    """
    cdef unsigned long    r = zran.zran_tell(&self.index)
    return r

zran_log showed that the desired value was calculated correctly and passed to zran_tell() for a return but when zran.zran_tell(&self.index) is called the returned value changed types from long to int which caused a negative output on Windows.

The description of a fresh clean working installation with Windows 64bit modifications is written in the previews post. Happy weekend and thanks .. :-)) Time to work with it!

[matthew-brett]

@cgohlke - do you have any interest in this package? (we're considering it as an optional dependency for nibabel). Can you help with the compilation? Thanks much.

[pauldmccarthy]

This is being addressed in #3 - hopefully I'll merge it in this weekend.

[pauldmccarthy]

While I haven't tested it myself (I don't have a windows machine), the unit tests are passing on appveyor, so I'm going to close this issue for the time being..