Crash in bunchkaufman test with LLVM 3.9.1

[nalimilan]

I get this crash locally since we moved to LLVM 3.9.1. It doesn't appear if I switch back to 3.7.1.

(Note that LLVM 3.9.1 incorrectly detects my CPU as skylake-avx512, while it only supports AVX2. Doesn't look like the cause of the crash, though, since fixing the CPU family and rebuilding LLVM and Julia doesn't fix it.)

julia> Base.runtests("linalg/bunchkaufman")
Test (Worker)           | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)

signal (11): Segmentation fault
while loading /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl, in expression starting on line 22
unknown function (ip: 0x7f7350df8a8c)
unknown function (ip: 0x7f7350e0014a)
unknown function (ip: 0x7f7350d8e648)
unknown function (ip: 0x7f7350d9593f)
unknown function (ip: 0x7f7350d95c37)
_ZN4llvm12RegAllocBase16allocatePhysRegsEv at /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so (unknown line)
unknown function (ip: 0x7f7350d96e4f)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so (unknown line)
operator() at /home/milan/Dev/julia/src/jitlayers.cpp:436
_M_invoke at /usr/include/c++/6.2.1/functional:1726
operator() at /usr/include/c++/6.2.1/functional:2136
addModuleSet<llvm::SmallVector<std::unique_ptr<llvm::Module>, 1u>, llvm::RTDyldMemoryManager*, std::unique_ptr<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> >, std::default_delete<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> > > > > at /home/milan/Dev/julia/usr/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h:73
addModule at /home/milan/Dev/julia/src/jitlayers.cpp:559
jl_add_to_ee at /home/milan/Dev/julia/src/jitlayers.cpp:782
jl_finalize_function at /home/milan/Dev/julia/src/jitlayers.cpp:793
getAddressForFunction at /home/milan/Dev/julia/src/codegen.cpp:1070
jl_generate_fptr at /home/milan/Dev/julia/src/codegen.cpp:1193
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:240
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:62 [inlined]
macro expansion; at ./test.jl:775 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:60 [inlined]
macro expansion; at ./test.jl:714 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:49 [inlined]
macro expansion; at ./test.jl:775 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:40 [inlined]
macro expansion; at ./test.jl:775 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:37 [inlined]
macro expansion; at ./test.jl:775 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/linalg/bunchkaufman.jl:24 [inlined]
macro expansion; at ./test.jl:775 [inlined]
anonymous at ./<missing> (unknown line)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_toplevel_eval_flex at /home/milan/Dev/julia/src/toplevel.c:648
jl_parse_eval_all at /home/milan/Dev/julia/src/ast.c:757
jl_load at /home/milan/Dev/julia/src/toplevel.c:675
jl_load_ at /home/milan/Dev/julia/src/toplevel.c:684
include_from_node1 at ./loading.jl:532
unknown function (ip: 0x7f73466793fb)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
include at ./sysimg.jl:14
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/testdefs.jl:13 [inlined]
macro expansion; at ./test.jl:714 [inlined]
macro expansion; at ./util.jl:288 [inlined]
macro expansion; at /home/milan/Dev/julia/usr/share/julia/test/testdefs.jl:0 [inlined]
anonymous at ./<missing> (unknown line)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_toplevel_eval_flex at /home/milan/Dev/julia/src/toplevel.c:648
jl_toplevel_eval at /home/milan/Dev/julia/src/toplevel.c:659
jl_toplevel_eval_in at /home/milan/Dev/julia/src/builtins.c:606
eval at ./boot.jl:236
unknown function (ip: 0x7f7346652c2f)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
runtests at /home/milan/Dev/julia/usr/share/julia/test/testdefs.jl:16
#529 at ./multi.jl:1053
run_work_thunk at ./multi.jl:1024
unknown function (ip: 0x7f7134ec10a9)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
#remotecall_fetch#534 at ./multi.jl:1078
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
jl_apply at /home/milan/Dev/julia/src/julia.h:1388
jl_f__apply at /home/milan/Dev/julia/src/builtins.c:548
remotecall_fetch at ./multi.jl:1078
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
jl_apply at /home/milan/Dev/julia/src/julia.h:1388
jl_f__apply at /home/milan/Dev/julia/src/builtins.c:548
#remotecall_fetch#538 at ./multi.jl:1106
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
jl_apply at /home/milan/Dev/julia/src/julia.h:1388
jl_f__apply at /home/milan/Dev/julia/src/builtins.c:548
remotecall_fetch at ./multi.jl:1106
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
macro expansion at /home/milan/Dev/julia/usr/share/julia/test/runtests.jl:65 [inlined]
#38 at ./task.jl:404
unknown function (ip: 0x7f7134ec0a8f)
jl_call_method_internal at /home/milan/Dev/julia/src/julia_internal.h:246
jl_apply_generic at /home/milan/Dev/julia/src/gf.c:2196
jl_apply at /home/milan/Dev/julia/src/julia.h:1388
start_task at /home/milan/Dev/julia/src/task.c:261
unknown function (ip: 0xffffffffffffffff)
Allocations: 10590801 (Pool: 10588840; Big: 1961); GC: 15
ERROR: A test has failed. Please submit a bug report (https://github.com/JuliaLang/julia/issues)
including error messages above and the output of versioninfo():
Julia Version 0.6.0-dev.1807
Commit 26c8d85* (2016-12-31 04:12 UTC)
DEBUG build
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake-avx512)

Stacktrace:
 [1] runtests(::String, ::Int64) at ./interactiveutil.jl:655
 [2] runtests(::String) at ./interactiveutil.jl:644

gdb backtrace:

julia> include("test/linalg/bunchkaufman.jl")

Thread 1 "julia-debug" received signal SIGSEGV, Segmentation fault.
0x00007ffff566da8d in llvm::SplitEditor::deleteRematVictims() ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
Missing separate debuginfos, use: dnf debuginfo-install http-parser-2.7.1-3.fc25.x86_64 libgcc-6.2.1-2.fc25.x86_64 libgfortran-6.2.1-2.fc25.x86_64 libquadmath-6.2.1-2.fc25.x86_64 libstdc++-6.2.1-2.fc25.x86_64 zlib-1.2.8-10.fc24.x86_64
(gdb) ba
#0  0x00007ffff566da8d in llvm::SplitEditor::deleteRematVictims() ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#1  0x00007ffff567514b in llvm::SplitEditor::finish(llvm::SmallVectorImpl<unsigned int>*) () from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#2  0x00007ffff5603649 in (anonymous namespace)::RAGreedy::doRegionSplit(llvm::LiveInterval&, unsigned int, bool, llvm::SmallVectorImpl<unsigned int>&) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#3  0x00007ffff560a940 in (anonymous namespace)::RAGreedy::selectOrSplitImpl(llvm::LiveInterval&, llvm::SmallVectorImpl<unsigned int>&, llvm::SmallSet<unsigned int, 16u, std::less<unsigned int> >&, unsigned int) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#4  0x00007ffff560ac38 in (anonymous namespace)::RAGreedy::selectOrSplit(llvm::LiveInterval&, llvm::SmallVectorImpl<unsigned int>&) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#5  0x00007ffff55f9d40 in llvm::RegAllocBase::allocatePhysRegs() ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#6  0x00007ffff560be50 in (anonymous namespace)::RAGreedy::runOnMachineFunction(llvm::MachineFunction&) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#7  0x00007ffff5575511 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) () from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#8  0x00007ffff5424442 in llvm::FPPassManager::runOnFunction(llvm::Function&)
    () from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
---Type <return> to continue, or q <return> to quit---
#9  0x00007ffff542475b in llvm::FPPassManager::runOnModule(llvm::Module&) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#10 0x00007ffff5424a77 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#11 0x00007ffff77ebaf1 in JuliaOJIT::<lambda(llvm::Module&)>::operator()(llvm::Module &) const (__closure=0x6e9368, M=...)
    at /home/milan/Dev/julia/src/jitlayers.cpp:436
#12 0x00007ffff77f8209 in std::_Function_handler<llvm::object::OwningBinary<llvm::object::ObjectFile>(llvm::Module&), JuliaOJIT::JuliaOJIT(llvm::TargetMachine&)::<lambda(llvm::Module&)> >::_M_invoke(const std::_Any_data &, llvm::Module &)
    (__functor=..., __args#0=...) at /usr/include/c++/6.2.1/functional:1726
#13 0x00007ffff77f966c in std::function<llvm::object::OwningBinary<llvm::object::ObjectFile> (llvm::Module&)>::operator()(llvm::Module&) const (this=0x6e9368, 
    __args#0=...) at /usr/include/c++/6.2.1/functional:2136
#14 0x00007ffff77f331f in llvm::orc::IRCompileLayer<llvm::orc::ObjectLinkingLayer<JuliaOJIT::DebugObjectRegistrar> >::addModuleSet<llvm::SmallVector<std::unique_ptr<llvm::Module>, 1u>, llvm::RTDyldMemoryManager*, std::unique_ptr<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> >, std::default_delete<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> > > > >(llvm::SmallVector<std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >, 1u>, llvm::RT---Type <return> to continue, or q <return> to quit---
DyldMemoryManager *, std::unique_ptr<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> >, std::default_delete<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> > > >) (this=0x6e9360, Ms=..., MemMgr=0x6f5cd0, 
    Resolver=std::unique_ptr<llvm::orc::LambdaResolver<JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)>, JuliaOJIT::addModule(std::unique_ptr<llvm::Module>)::<lambda(const string&)> >> containing 0x3807110)
    at /home/milan/Dev/julia/usr/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h:73
#15 0x00007ffff77ec711 in JuliaOJIT::addModule (this=0x6e8100, 
    M=std::unique_ptr<llvm::Module> containing 0x0)
    at /home/milan/Dev/julia/src/jitlayers.cpp:559
#16 0x00007ffff77ed267 in jl_add_to_ee (
    m=std::unique_ptr<llvm::Module> containing 0x0)
    at /home/milan/Dev/julia/src/jitlayers.cpp:782
#17 0x00007ffff77ed33f in jl_finalize_function (F=0x4137a58)
    at /home/milan/Dev/julia/src/jitlayers.cpp:793
#18 0x00007ffff7773e93 in getAddressForFunction (llvmf=0x4137a58)
    at /home/milan/Dev/julia/src/codegen.cpp:1070
#19 0x00007ffff77744b0 in jl_generate_fptr (li=0x7ffdef82cf90, _F=0x4137a58, 
    world=20528) at /home/milan/Dev/julia/src/codegen.cpp:1193
---Type <return> to continue, or q <return> to quit---
#20 0x00007ffff76eac92 in jl_call_method_internal (meth=0x7ffdef82cf90, 
    args=0x7fffffffb370, nargs=3)
    at /home/milan/Dev/julia/src/julia_internal.h:240
#21 0x00007ffff76f2655 in jl_apply_generic (args=0x7fffffffb370, nargs=3)
    at /home/milan/Dev/julia/src/gf.c:2196
#22 0x00007ffdd9729c47 in ?? ()
#23 0x00007fffffffc200 in ?? ()
#24 0x0000000000005030 in ?? ()
#25 0x00007ffdeff776d0 in ?? ()
#26 0x00007ffdeff77fa0 in ?? ()
#27 0x00007ffde91997b0 in ?? ()
#28 0x00007ffde8a075b0 in ?? ()
#29 0x00007ffde89fa890 in ?? ()
#30 0x00007ffde8a12170 in ?? ()
#31 0x00007ffde8a121d0 in ?? ()
#32 0x00007ffdefbe5530 in ?? ()
#33 0x00007ffde89d9870 in ?? ()
#34 0x0000000000000002 in ?? ()
#35 0x00007ffde8ab4940 in ?? ()
#36 0x00007ffde902db78 in ?? ()
#37 0x00007ffde9d16450 in ?? ()
#38 0x00007ffdecc2b4f8 in ?? ()
#39 0x00007ffdf110b940 in ?? ()
---Type <return> to continue, or q <return> to quit---
#40 0x00007ffdefcab2b0 in ?? ()
#41 0x0000000000000001 in ?? ()
#42 0x00007ffdf0239ef0 in ?? ()
#43 0x0000000000000001 in ?? ()
#44 0x00007ffdf0238fd0 in ?? ()
#45 0x00007ffde902f5d0 in ?? ()
#46 0x0000000000000002 in ?? ()
#47 0x00007ffde8989070 in ?? ()
#48 0x00007ffde8989080 in ?? ()
#49 0x0000000000000003 in ?? ()
#50 0x00007ffdec2e9b80 in ?? ()
#51 0x00007ffdecc29df0 in ?? ()
#52 0x00007ffde902cbd0 in ?? ()
#53 0x00007ffdeb527ec0 in ?? ()
#54 0x00007ffdeb559b80 in ?? ()
#55 0x0000000003943205 in ?? ()
#56 0x00007fffffff9d08 in ?? ()
#57 0x00007fffffff9fd0 in ?? ()
#58 0x00007ffff57dc0a2 in llvm::RegsForValue::getCopyFromRegs(llvm::SelectionDAG&, llvm::FunctionLoweringInfo&, llvm::SDLoc const&, llvm::SDValue&, llvm::SDValue*, llvm::Value const*) const ()
   from /home/milan/Dev/julia/usr/bin/../lib/libLLVM-3.9.so
#59 0x00007ffde8960df0 in ?? ()
---Type <return> to continue, or q <return> to quit---
#60 0x00007ffde89bbf70 in ?? ()
#61 0x00007ffdeb559cb8 in ?? ()
#62 0x0000000000000011 in ?? ()
#63 0x00007ffdeb5e30d0 in ?? ()
#64 0x00007ffde8960cd0 in ?? ()
#65 0x00007ffdee8e5c60 in ?? ()
#66 0x010100ffffff9e80 in ?? ()
#67 0x00007ffff7719009 in jl_unprotect_stack (F=0x7ffde8960cd0, 
    args=0x7ffdeb5e30d0, nargs=0) at /home/milan/Dev/julia/src/task.c:648
#68 0x0000000000737400 in ?? ()
#69 0x00007ffded778430 in ?? ()
#70 0x00007ffde88b8718 in ?? ()
#71 0x00007ffff7702441 in jl_eqtable_put (h=0x11, key=0x7ffdeb559cb8, 
    val=0x7ffde89bbf70) at /home/milan/Dev/julia/src/table.c:136
#72 0x00007ffde8980060 in ?? ()
#73 0x00007ffde902cb50 in ?? ()
#74 0x00007ffde8989070 in ?? ()
#75 0x00007ffde8965870 in ?? ()
#76 0x00007ffff7fcd1c8 in ?? ()
#77 0x01000100ffff9ec9 in ?? ()
#78 0x0000000000000594 in ?? ()
#79 0x00007fffffffc270 in ?? ()
#80 0x00007ffdf0c10d30 in ?? ()
---Type <return> to continue, or q <return> to quit---
#81 0x00007ffdeff77fa0 in ?? ()
#82 0x00007ffde8a075b0 in ?? ()
#83 0x00007ffde8a075b0 in ?? ()
#84 0x00007ffde8989080 in ?? ()
#85 0x00007ffde8989070 in ?? ()
#86 0x0000000000000000 in ?? ()

[tkelman]

very likely duplicate of #19792

[tkelman]

closing as dup

[vchuravy]

(Note that LLVM 3.9.1 incorrectly detects my CPU as skylake-avx512, while it only supports AVX2. Doesn't look like the cause of the crash, though, since fixing the CPU family and rebuilding LLVM and Julia doesn't fix it.)

Has this been reported upstream? We either need to fix this on our side or patch it in LLVM

[nalimilan]

I'd like to report it upstream, but automated account creation is suspended on their Bugzilla, and they haven't replied to my e-mail yet. If you want to do it for me, here is my cpuinfo; the problem is simply that the 78 (i.e. 0x4e) family is considered as skylake-avx512 in lib/Support/Host.cpp. (It's also funny that all CPUs are i7 there, but I guess it's just a matter of naming.)

[vchuravy]

Can you check if:

--- a/lib/Support/Host.cpp
+++ b/lib/Support/Host.cpp
@@ -476,7 +476,11 @@ getIntelProcessorTypeAndSubtype(unsigned int Family, unsigned int Model,
     // Skylake:
     case 0x4e:
       *Type = INTEL_COREI7; // "skylake-avx512"
-      *Subtype = INTEL_COREI7_SKYLAKE_AVX512;
+      if (Features & (1 << FEATURE_AVX512)) {
+        *Subtype = INTEL_COREI7_SKYLAKE_AVX512;
+      } else {
+        *Subtype = INTEL_COREI7_SKYLAKE;
+      }
       break;
     case 0x5e:
       *Type = INTEL_COREI7; // "skylake"

is enough?

[nalimilan]

Yes, that works for my CPU, and that sounds like the obvious thing to do.

(Just curious: isn't anybody around here using a desktop Skylake CPU (without AVX512)? I would have thought it would be relatively common.)

[vchuravy]

Surprisingly skylake is split into two normal Skylake and Skylake Xeon (which is a server variant that supports AVX512) your CPU uses the model I'd normally used by skx

[vchuravy]

Ok opened a revision with LLVM https://reviews.llvm.org/D28221 @Keno, @yuyichao can you take a look so that we might add this before v0.6?

[tkelman]

seems fine to carry now? I can add it to #19810

[Keno]

Yes, fine to carry

[tkelman]

oops, looks like you changed the skylake avx512 patch again for some upstream review after I committed it here. we can adjust it to match the final upstream version whenever that happens.