"LLVM ERROR: Broken module found, compilation aborted!" when compiling a sysimage with a ccallable function and a certain cpu target

[KristofferC]

When trying to change the cpu target used when generating "apps" in PackageCompiler (JuliaLang/PackageCompilerX.jl#43) with the "app" at (https://github.com/KristofferC/PackageCompilerX.jl/blob/master/examples/MyApp/src/MyApp.jl) I hit the following CI error

Linux (https://travis-ci.com/KristofferC/PackageCompilerX.jl/jobs/265321335#L264-L267):

<unknown>: error: Cannot represent a difference across sections
Aliasee should be either GlobalValue or ConstantExpr
i32 ()* @julia_main
LLVM ERROR: Broken module found, compilation aborted!

Mac (https://travis-ci.com/KristofferC/PackageCompilerX.jl/jobs/265321336#L197-L212);

signal (11): Segmentation fault: 11
in expression starting at none:0
_ZN12_GLOBAL__N_119X86MachObjectWriter16recordRelocationEPN4llvm16MachObjectWriterERNS1_11MCAssemblerERKNS1_11MCAsmLayoutEPKNS1_10MCFragmentERKNS1_7MCFixupENS1_7MCValueERy at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm16MachObjectWriter16recordRelocationERNS_11MCAssemblerERKNS_11MCAsmLayoutEPKNS_10MCFragmentERKNS_7MCFixupENS_7MCValueERy at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm11MCAssembler6layoutERNS_11MCAsmLayoutE at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm11MCAssembler6FinishEv at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN12_GLOBAL__N_115MCMachOStreamer10FinishImplEv at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm10MCStreamer6FinishEv at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /Users/travis/julia/lib/julia/libLLVM.dylib (unknown line)
operator() at /Users/sabae/buildbot/worker/package_macos64/build/src/jitlayers.cpp:1086 [inlined]
jl_dump_native at /Users/sabae/buildbot/worker/package_macos64/build/src/jitlayers.cpp:1095
jl_write_compiler_output at /Users/sabae/buildbot/worker/package_macos64/build/src/precompile.c:93
jl_atexit_hook at /Users/sabae/buildbot/worker/package_macos64/build/src/init.c:230
main at /Users/travis/julia/bin/julia (unknown line)

It can be reprod on the linked branch above with pwd in the root git repo and running:

using PackageCompilerX
 create_sysimage(:MyApp, sysimage_path="MyApp.so", cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)", project="examples/MyApp", incremental=false)

Note that incremental=false seems to be needed (which means that a new sysimage is bootstrapped from scratch (corecompiler.ji -> sys.ji -> sys.o).

I can reduce it to just a set of raw commands if that would be helpful.

[KristofferC]

On the backport 1.3.1 branch I got

Assertion failed: (isa<X>(Val) && "cast<Ty>() argument of incompatible type!"), function cast, file /Users/kristoffer/julia/usr/include/llvm/Support/Casting.h, line 255.

[KristofferC]

Here is an MWE without PackageCompilerX. It seems that the combination for this to occur is:

• use sys.ji as sysimage
• use the cpu target we use for the binaries
• compile a Base.@ccallable function

function get_julia_cmd()
    julia_path = joinpath(Sys.BINDIR, Base.julia_exename())
    cmd = `$julia_path --color=yes --startup-file=no`
end

sysimg_source_path = Base.find_source_file("sysimg.jl")
base_dir = dirname(sysimg_source_path)
tmp_corecompiler_ji = joinpath(@__DIR__, "corecompiler.ji")
tmp_sys_ji = joinpath(@__DIR__, "sys.ji")
compiler_source_path = joinpath(base_dir, "compiler", "compiler.jl")
cd(base_dir) do
    # Create corecompiler.ji
    cmd = `$(get_julia_cmd()) --output-ji $tmp_corecompiler_ji
                              -g0 -O0 $compiler_source_path`
    run(cmd)

    # Use that to create sys.ji
    cmd = `$(get_julia_cmd()) --sysimage=$tmp_corecompiler_ji
                              -g1 -O0 --output-ji=$tmp_sys_ji $sysimg_source_path`
    run(cmd)
end

app_content = """
Base.@ccallable function julia_main()::Cint
    return 0
end
"""

# Create the sysimage
cmd = `$(get_julia_cmd()) --sysimage=$tmp_sys_ji --cpu-target='generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)'
                          --output-o=sys.o -e $app_content`
run(cmd)

[KristofferC]

A workaround is to first create an empty dylib sysimage from sys.ji and then incrementally add the @ccallable function to that sysimage. Works but takes a bit longer time than neeed.

[KristofferC]

Still failing, now with:

julia: /home/kc/julia/usr/include/llvm/Support/Casting.h:264: typename llvm::cast_retty<X, Y*>::ret_type llvm::cast(Y*) [with X = llvm::GlobalVariable; Y = llvm::GlobalValue; typename llvm::cast_retty<X, Y*>::ret_type = llvm::GlobalVariable*]: Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

signal (6): Aborted
in expression starting at none:0
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fb078b17728)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
cast<llvm::GlobalVariable, llvm::GlobalValue> at /home/kc/julia/usr/include/llvm/Support/Casting.h:264 [inlined]
operator() at /home/kc/julia/src/llvm-multiversioning.cpp:711 [inlined]
fix_gv_uses at /home/kc/julia/src/llvm-multiversioning.cpp:726 [inlined]
runOnModule at /home/kc/julia/src/llvm-multiversioning.cpp:1053
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/kc/julia/usr/bin/../lib/libLLVM-9jl.so (unknown line)
operator() at /home/kc/julia/src/aotcompile.cpp:534 [inlined]
jl_dump_native at /home/kc/julia/src/aotcompile.cpp:545
jl_write_compiler_output at /home/kc/julia/src/precompile.c:90
jl_atexit_hook at /home/kc/julia/src/init.c:218
main at /home/kc/julia/ui/repl.c:228
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/kc/julia/usr/bin/julia (unknown line)
Allocations: 15518740 (Pool: 15510414; Big: 8326); GC: 3

[vtjnash]

Possibly we might need https://github.com/JuliaLang/julia/tree/jn/localize-multiversioning for that (though it doesn't currently include the fix)

[vchuravy]

Seeing this in #37510 in various forms:

    JULIA /home/vchuravy/builds/julia/usr/lib/julia/sys-o.a
<unknown>:0: error: Cannot represent a difference across sections
<unknown>:0: error: Cannot represent a difference across sections
<unknown>:0: error: Cannot represent a difference across sections
<unknown>:0: error: Cannot represent a difference across sections
<unknown>:0: error: Cannot represent a difference across sections
<unknown>:0: error: Cannot represent a difference across sections
Aliasee should be either GlobalValue or ConstantExpr
i16 (float)* @__truncsfhf2
Aliasee should be either GlobalValue or ConstantExpr
float (i16)* @__gnu_h2f_ieee
Aliasee should be either GlobalValue or ConstantExpr
i16 (float)* @__gnu_f2h_ieee
Aliasee should be either GlobalValue or ConstantExpr
float (i16)* @__extendhfdf2
Aliasee should be either GlobalValue or ConstantExpr
i16 (double)* @__truncdfhf2
Aliasee should be either GlobalValue or ConstantExpr
float (i16)* @__extendhfsf2
LLVM ERROR: Broken module found, compilation aborted!

and

4:44
julia: /buildworker/worker/package_linux32/build/usr/include/llvm/Support/Casting.h:264: typename llvm::cast_retty<X, Y*>::ret_type llvm::cast(Y*) [with X = llvm::GlobalVariable; Y = llvm::GlobalValue; typename llvm::cast_retty<X, Y*>::ret_type = llvm::GlobalVariable*]: Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
signal (6): Aborted
in expression starting at none:0
gsignal at /lib/i386-linux-gnu/libc.so.6 (unknown line)
abort at /lib/i386-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0xf7a830f5)
__assert_fail at /lib/i386-linux-gnu/libc.so.6 (unknown line)
cast<llvm::Function, llvm::Value> at /buildworker/worker/package_linux32/build/usr/include/llvm/Support/Casting.h:264 [inlined]
emit_metadata at /buildworker/worker/package_linux32/build/src/llvm-multiversioning.cpp:999 [inlined]
runOnModule at /buildworker/worker/package_linux32/build/src/llvm-multiversioning.cpp:1068
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /buildworker/worker/package_linux32/build/usr/bin/../lib/libLLVM-9jl.so (unknown line)
_ZN4llvm6legacy11PassManager3runERNS_6ModuleE at /buildworker/worker/package_linux32/build/usr/bin/../lib/libLLVM-9jl.so (unknown line)
operator() at /buildworker/worker/package_linux32/build/src/aotcompile.cpp:550 [inlined]
jl_dump_native at /buildworker/worker/package_linux32/build/src/aotcompile.cpp:561
jl_write_compiler_output at /buildworker/worker/package_linux32/build/src/precompile.c:94
jl_atexit_hook at /buildworker/worker/package_linux32/build/src/init.c:210
main at /buildworker/worker/package_linux32/build/ui/repl.c:228
__libc_start_main at /lib/i386-linux-gnu/libc.so.6 (unknown line)
_start at /buildworker/worker/package_linux32/build/usr/bin/julia (unknown line)
Allocations: 116324446 (Pool: 116309614; Big: 14832); GC: 222
Aborted (core dumped)

[KristofferC]

Looks like the same thing since that PR also uses @ccallable.