Flux fails to precompile on v0.7 and macOS

[JBlaschke]

On mac0S 10.13.6 i get:

julia> import Flux
[ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
ERROR: LoadError: LoadError: error compiling top-level scope: could not load library "libz"
dlopen(libz.dylib, 1): image not found

@SimonDanisch recommends using https://github.com/bicycle1885/CodecZlib.jl
see: JuliaIO/GZip.jl#54 (comment)

[JBlaschke]

Just and FYI: this issue persists for Julia v1.0 also...

[jumerckx]

I had the same problem on my Ubuntu 18.04 laptop. The problem got resolved once I installed zlib:
sudo apt-get install zlib1g-dev
Hope this helps,
Jules

[JBlaschke]

I have zlib already installed (via Homebrew I think). And I am able to see libz.dylib. The problem is that Julia does not find it.

[jonasaugust]

@merckxiaan Installing zlib1g-dev as you did also works around this issue on Ubuntu 16.04 amd64 for Julia 1.0.0. Thanks.

[JBlaschke]

I have finally found a (very annoying) workaround for macOS. As I said earlier, the problem wasn't that zlib isn't installed (hence the suggestions to install zlib1g-dev doesn't actually address the issue), but that macOS' dynamic linker can't find it.

A temporary solution is to set the dynamic linker's library path:

set -x DYLD_LIBRARY_PATH /usr/lib/

in fish or

export DYLD_LIBRARY_PATH=/usr/lib/

in bash

This is very much a hack, and will allow Julia to compile GZip.jl and Flux.jl. But it will crash the Julia interpreter if you try anything else (since you're messing with other linking systems in Julia). So my solution is to set -x DYLD_LIBRARY_PATH /usr/lib/ -> precompile Flux -> unset DYLD_LIBRARY_PATH -> restart Julia.

This is very much a temporary solution, and I would suggest keeping up the pressure in the GZip.jl people so they come up with a permanent solution.

[MikeInnes]

@JBlaschke, or anyone else with this issue, can you confirm that this is fixed on Flux master? If so I'll tag release.

[JBlaschke]

@MikeInnes Thanks! I just tried a ] update, but I assume your changes aren't in Julia's registry yet because it's still trying to install GZip... Or am I missing something here/how does one add a package straight from github?

[SimonDanisch]

tagging a release -> having it in the registry ;) @MikeInnes is asking you to do:
]add Flux#master to test if it's fixed before tagging, to make sure that ]update doesn't just install another faulty version ;)
If you've done ]dev Flux before, don't forget to call ]free Flux

[JBlaschke]

@SimonDanisch and @MikeInnes thanks for the info. The version in master compiles fine now. Thank you!

[MikeInnes]

Great. Release coming shortly.

[mbeltagy]

I have a fresh Julia 1.01 install with flux and I get the a similar error

julia> using Flux, CuArrays
[ Info: Recompiling stale cache file /home/elbeltagy/.julia/compiled/v1.0/Flux/QdkVy.ji for Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
WARNING: could not import NNlib.cudata into Tracker
Exception handling log message: ErrorException("error compiling #handle_message#2: error compiling write: could not load library \"libz\"\nlibz.so: cannot open shared object file: No such file or directory")
  module=Base  file=loading.jl  line=1187
  Second exception: ErrorException("error compiling #handle_message#2: error compiling write: could not load library \"libz\"\nlibz.so: cannot open shared object file: No such file or directory")
Exception handling log message: ErrorException("error compiling #handle_message#2: error compiling write: could not load library \"libz\"\nlibz.so: cannot open shared object file: No such file or directory")
  module=Requires  file=/home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl  line=40
  Second exception: ErrorException("error compiling #handle_message#2: error compiling write: could not load library \"libz\"\nlibz.so: cannot open shared object file: No such file or directory")

If I do

sudo apt-get install zlib1g-dev

as @merckxiaan suggested the libz errors go away...but then I now have a new set of warnings

julia> using Flux, CuArrays
┌ Warning: Error requiring CuArrays from Flux:
│ LoadError: UndefVarError: cudnn_available not defined
│ Stacktrace:
│  [1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
│  [2] top-level scope at none:0
│  [3] include at ./boot.jl:317 [inlined]
│  [4] include_relative(::Module, ::String) at ./loading.jl:1041
│  [5] include at ./sysimg.jl:29 [inlined]
│  [6] include(::String) at /home/elbeltagy/.julia/packages/Flux/UHjNa/src/Flux.jl:1
│  [7] top-level scope at none:0
│  [8] eval at ./boot.jl:319 [inlined]
│  [9] eval at /home/elbeltagy/.julia/packages/Flux/UHjNa/src/Flux.jl:1 [inlined]
│  [10] (::getfield(Flux, Symbol("##99#103")))() at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:67
│  [11] err(::getfield(Flux, Symbol("##99#103")), ::Module, ::String) at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:38
│  [12] #98 at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:66 [inlined]
│  [13] withpath(::getfield(Flux, Symbol("##98#102")), ::String) at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:28
│  [14] (::getfield(Flux, Symbol("##97#101")))() at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:65
│  [15] #invokelatest#1 at ./essentials.jl:697 [inlined]
│  [16] invokelatest at ./essentials.jl:696 [inlined]
│  [17] #3 at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:19 [inlined]
│  [18] iterate at ./generator.jl:47 [inlined]
│  [19] collect_to!(::Array{Function,1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64, ::Int64) at ./array.jl:656
│  [20] collect_to!(::Array{typeof(CUDAnative.cudaconvert),1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64, ::Int64) at ./array.jl:667
│  [21] collect_to_with_first!(::Array{typeof(CUDAnative.cudaconvert),1}, ::Function, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Int64) at ./array.jl:643
│  [22] _collect(::Array{Function,1}, ::Base.Generator{Array{Function,1},getfield(Requires, Symbol("##3#4"))}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:637
│  [23] map at ./array.jl:561 [inlined]
│  [24] loadpkg(::Base.PkgId) at /home/elbeltagy/.julia/packages/Requires/9Jse8/src/require.jl:19
│  [25] #invokelatest#1 at ./essentials.jl:697 [inlined]
│  [26] invokelatest at ./essentials.jl:696 [inlined]
│  [27] require(::Base.PkgId) at ./loading.jl:858
│  [28] macro expansion at ./logging.jl:311 [inlined]
│  [29] require(::Module, ::Symbol) at ./loading.jl:837
│  [30] eval(::Module, ::Any) at ./boot.jl:319
│  [31] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
│  [32] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]
│  [33] (::getfield(REPL, Symbol("##28#29")){REPL.REPLBackend})() at ./task.jl:259
│ in expression starting at /home/elbeltagy/.julia/packages/Flux/UHjNa/src/cuda/cuda.jl:5
└ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40

I have cuda 9.0 installed and also libcudnn 7.1.4

[mbeltagy]

My current problem now is related #420 ... Which I hope should be fixed in the next release.

However the problem related missing libz is still an issues on linux install with the following

[3a865a2d] CuArrays v0.8.1
[587475ba] Flux v0.6.7

[MikeInnes]

The latest Flux is v0.6.8, so try updating your packages. If you still have the zlib issue you might want to report it to CodecZlib.jl