ABI conflicts due to 64-bit libopenblas.so

[stevengj]

Julia compiles OpenBLAS to libopenblas.so. This may be a problem for calling libraries that link to a system libopenblas.so, because the runtime linker may substitute Julia's version instead. The problem is that Julia's version is compiled with a 64-bit interface, which is not the default, and so if an external library calls it expecting a 32-bit interface, a crash may result.

We encountered what appears to have been this problem n @alanedelman's machine (julia.mit.edu). He recently started experiencing crashes in PyPlot.plot that, with the help of valgrind, I tracked down to apparently:

==17855== Use of uninitialised value of size 8
==17855==    at 0xA8B6890: dgemm_beta_NEHALEM (in /home/edelman/julia/usr/lib/libopenblas.so)
==17855==    by 0xA082D72: dgemm_nn (in /home/edelman/julia/usr/lib/libopenblas.so)
==17855==    by 0x9F558C8: cblas_dgemm (in /home/edelman/julia/usr/lib/libopenblas.so)
==17855==    by 0x16430CA5: dotblas_matrixproduct (_dotblas.c:809)
==17855==    by 0x14BAB5D4: PyEval_EvalFrameEx (in /usr/lib/libpython2.7.so.1.0)

Apparently, Matplotlib is calling OpenBLAS (via NumPy: _dotblas.c is a NumPy file) with the 32-bit interface, but is getting linked at runtime into Julia's openblas library, which is compiled with a 64-bit interface. Recompiling Julia and openblas with USE_BLAS64=0 worked around the problem, but it would be better to avoid the conflict.

Can we just rename our libopenblas.so file to avoid any possible conflict in the runtime linker?

[stevengj]

Or is the problem worse than that? If I ccall a library that in turn calls cblas_dgemm, will it end up calling our OpenBLAS version even if it was originally linked to a completely different BLAS library (e.g. libblas.so)?

In that case, we might have to hack OpenBLAS to rename its exported functions (e.g. cblas_dgemm64 etcetera) since we changed the ABI.

[stevengj]

@xianyi, is there a way to tell OpenBLAS to add a prefix or suffix (e.g. 64) to all its exported symbols, to make it possible to link both the 32-bit and 64-bit ABI in the same executable?

[stevengj]

See also numpy/numpy#3916

[StefanKarpinski]

Wouldn't it make more sense to put the 64 after the cblas part – as in cblas64_dgemm?

[ViralBShah]

The ideal solution would be to have a separate 64-bit ABI and build both 32 and 64 bit versions in the same library.

[staticfloat]

@ViralBShah that is actually the best solution here. That would be wonderful!

[stevengj]

@StefanKarpinski, note that there is a Fortran dgemm ABI too, and to avoid conflicts you need to rename both C and Fortran (unless we are not linking the Fortran ABI?). But I don't think it really matters what the name looks like, as long as there is a simple deterministic rule and it can be implemented as automatically as possible in the openblas source code. I was just thinking that a suffix might be easier to automate for both C and Fortran ABIs.

[ViralBShah]

Currently we use the fortran abi only.

[ViralBShah]

I wonder if we can somehow make matplotlib use its own blas. While we may be able to do all sorts of gymnastics with openblas, it will be difficult to do the same with vendor provided BLAS.

[stevengj]

The other alternative would be to recompile our own numpy, but that makes installing PyCall much more of pain.

[stevengj]

@ViralBShah, does MKL provide the 64-bit ABI?

[StefanKarpinski]

The other alternative would be to recompile our own numpy, but that makes installing PyCall much more of pain.

Not to mention that the amount of stuff we compile ourselves is getting slightly ridiculous. But it's hard to avoid.

[ViralBShah]

I believe MKL does have a 64-bit ABI - but not 100% sure. @andreasnoackjensen ?