Unreliable SIGINT delivery

[maleadt]

Calling kill(self, 2) in 0.5 is unreliable:

ccall(:kill, Void, (Cint, Cint,), getpid(), 2)

Running this snippet results in different possible errors:

signal (2): Interrupt
while loading no file, in expression starting on line 0
anonymous at ./<missing> (unknown line)
jl_call_method_internal at /home/tbesard/Projects/Julia/src/julia_internal.h:178 [inlined]
jl_toplevel_eval_flex at /home/tbesard/Projects/Julia/src/toplevel.c:569
jl_parse_eval_all at /home/tbesard/Projects/Julia/src/ast.c:717
jl_load at /home/tbesard/Projects/Julia/src/toplevel.c:596 [inlined]
jl_load_ at /home/tbesard/Projects/Julia/src/toplevel.c:605

signal (2): Interrupt
while loading no file, in expression starting on line 0
jl_unbox_int64 at /home/tbesard/Projects/Julia/src/alloc.c:1179
jl_f_getfield at /home/tbesard/Projects/Julia/src/builtins.c:689

signal (2): Interrupt
while loading no file, in expression starting on line 0

signal (11): Segmentation fault
while loading no file, in expression starting on line 0

None of them properly delivering the SIGINT.

Meanwhile, on 0.4:

$ julia crash.jl
$ echo $?
130     # indicates SIGINT

This is a reduced test-case from core.jl, which in its current form it only fails when running under ASAN:

function crash()
    ccall(:jl_exit_on_sigint, Void, (Cint,), 0)
    try
        ccall(:kill, Void, (Cint, Cint,), getpid(), 2)
        for i in 1:10
            Libc.systemsleep(0.1)
            ccall(:jl_gc_safepoint, Void, ()) # wait for SIGINT to arrive
        end
    catch ex
        isa(ex, InterruptException) || rethrow(ex)
    end
    ccall(:jl_exit_on_sigint, Void, (Cint,), 1)
end

crash()

ASAN:DEADLYSIGNAL
=================================================================
==17111==ERROR: AddressSanitizer: SEGV on unknown address 0x7fed15b90000 (pc 0x7fecf6cc140e bp 0x7ffe17280530 sp 0x7ffe172803b0 T0)
    #0 0x7fecf6cc140d  (<unknown module>)
    #1 0x7fecf6cc154f  (<unknown module>)
    #2 0x7fed244605d0 in jl_call_method_internal /home/tbesard/Projects/Julia/src/julia_internal.h:178:16
    #3 0x7fed244694c2 in jl_apply_generic /home/tbesard/Projects/Julia/src/gf.c:1928:23
    #4 0x7fed244bf131 in do_call /home/tbesard/Projects/Julia/src/interpreter.c:66:26
    #5 0x7fed244b5cbf in eval /home/tbesard/Projects/Julia/src/interpreter.c:190:16
    #6 0x7fed244b50f1 in jl_interpret_toplevel_expr /home/tbesard/Projects/Julia/src/interpreter.c:31:12
    #7 0x7fed2451d436 in jl_toplevel_eval_flex /home/tbesard/Projects/Julia/src/toplevel.c:558:26
    #8 0x7fed2448fc71 in jl_parse_eval_all /home/tbesard/Projects/Julia/src/ast.c:717:26
    #9 0x7fed2451f970 in jl_load /home/tbesard/Projects/Julia/src/toplevel.c:596:12
    #10 0x7fed2451fb9a in jl_load_ /home/tbesard/Projects/Julia/src/toplevel.c:605:23
    #11 0x7fed13335361 in julia_include_from_node1_19006 loading.jl:426
    #12 0x7fed1333552b in jlcall_include_from_node1_19006 (/home/tbesard/Projects/Julia/build/sanitize/usr/lib/julia/sys-debug.so+0x30952b)
    #13 0x7fed244605d0 in jl_call_method_internal /home/tbesard/Projects/Julia/src/julia_internal.h:178:16
    #14 0x7fed244694c2 in jl_apply_generic /home/tbesard/Projects/Julia/src/gf.c:1928:23
    #15 0x7fed1336b061 in julia_process_options_20109 client.jl:262
    #16 0x7fed1336d5ea in julia__start_20103 client.jl:318
    #17 0x7fed1336e178 in jlcall__start_20103 (/home/tbesard/Projects/Julia/build/sanitize/usr/lib/julia/sys-debug.so+0x342178)
    #18 0x7fed244605d0 in jl_call_method_internal /home/tbesard/Projects/Julia/src/julia_internal.h:178:16
    #19 0x7fed244694c2 in jl_apply_generic /home/tbesard/Projects/Julia/src/gf.c:1928:23
    #20 0x4dd344 in jl_apply /home/tbesard/Projects/Julia/ui/../src/julia.h:1392:12
    #21 0x4dcb3c in true_main /home/tbesard/Projects/Julia/ui/repl.c:112:9
    #22 0x4dc9ad in main /home/tbesard/Projects/Julia/ui/repl.c:232:15
    #23 0x7fed23134740 in __libc_start_main (/usr/lib/libc.so.6+0x20740)
    #24 0x4190c8 in _start (/home/tbesard/Projects/Julia/build/sanitize/usr/bin/julia-debug+0x4190c8)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>) 
==17111==ABORTING

I'm not sure what's happening, but there seems to be an issue with the SIGINT delivery. If I transform the above test-case, ASAN sometimes spits out more information. In those cases, it traps a use-after-free on a known address, with those addresses always being freed by some trace starting at __cxa_finalize. So it looks like the SIGUSR2 thrown by jl_try_deliver_sigint causes the world to collapse?

cc @yuyichao

[JeffBezanson]

👍

I've noticed a few other issues with interrupting/signals, maybe related, maybe not. For example:

julia> while true
       end
^C^C^C^C^C^CWARNING: Force throwing a SIGINT
^C^Cfatal: error thrown and no exception handler available.
InterruptException()
jl_mutex_unlock at /home/jeff/src/julia/src/./julia_threads.h:478 [inlined]
jl_compile_for_dispatch at /home/jeff/src/julia/src/gf.c:1314
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:173 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
with_output_color at ./util.jl:232
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
_start at ./client.jl:364
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/ui/../src/julia.h:1392 [inlined]
true_main at /home/jeff/src/julia/ui/repl.c:112
main at /home/jeff/src/julia/ui/repl.c:232
# julia exits!

It would be nice if we could interrupt while true with a single ^C (as in 0.4). In 0.5 you need to hit it repeatedly and use the force-throw mechanism. I could live with that, except then the julia process exits (with code 130).

[JeffBezanson]

Also maybe related: #17626

[yuyichao]

The segfault is expected and is how the signal is delivered. The while true is why the force throw mechanism is added.

[yuyichao]

And the unreliable-ness is due to multithread support.

[JeffBezanson]

The while true is why the force throw mechanism is added.

But why does the process exit? I'm pretty sure it didn't used to do that, even with force throw.

[yuyichao]

So in summary, #12904 makes the signal handling unreliable for exit_on_sigint since the jl_exit is running on the signal handling thread/in the signal handler for that case which is either unmanaged or have a smaller stack. The sigint safepoint PR makes the !exit_on_sigint case more reliable (i.e.

ccall(:jl_exit_on_sigint, Void, (Cint,), 0)
ccall(:kill, Void, (Cint, Cint,), getpid(), 2)

should work) but doesn't fix the exit_on_sigint case. Fixing the exit_on_sigint case requires writing some platform dependent stack manipulation to make sure jl_exit always run on the right thread, which is why I added it to 0.6.0. It can possibly be 0.5.x if it turns out to be not too breaking (i.e. if the change is limited to a few files).

But why does the process exit? I'm pretty sure it didn't used to do that, even with force throw.

It's likely because of missing try-catch in the REPL code. I've seen it but can't reproduce it right now so I can't tell where it throws. If we can find where that happens, a try-catch and defer_sigint should work.

[JeffBezanson]

Does the stack trace I posted above help? It looks like it throws from the catch block of _start, presumably after catching the first InterruptException.

[yuyichao]

Does this help?

diff --git a/base/client.jl b/base/client.jl                                      
index 19cc480..04bf114 100644                                                     
--- a/base/client.jl                                                              
+++ b/base/client.jl                                                              
@@ -314,7 +314,9 @@ function _start()                                             
     empty!(ARGS)                                                                 
     append!(ARGS, Core.ARGS)                                                     
     opts = JLOptions()                                                           
+    ccall(:jl_sigatomic_begin, Void, ())                                         
     try                                                                          
+        ccall(:jl_sigatomic_end, Void, ())                                       
         (quiet,repl,startup,color_set,history_file) = process_options(opts)      

         local term                                                               
@@ -364,6 +366,7 @@ function _start()                                             
         display_error(err,catch_backtrace())                                     
         exit(1)                                                                  
     end                                                                          
+    ccall(:jl_sigatomic_end, Void, ())                                           
     if is_interactive && have_color                                              
         print(color_normal)                                                      
     end                                                                          

[yuyichao]

Although it seems that you've got a throw from another inner catch since it's already "bad" when we reach this catch (it's going to exit next)

[yuyichao]

OTOH, making this transformation automatically is possible and will make all catch block sigatomic. However, I currently prefer to only add that manually for a few ones in the REPL code since doing that automatically will make it much harder (has to force throw) to break out of error handling/printing code, which is more dangerous than the current force throw since it is currently only used on loops without safepoint (usually safe) or buggy C code (problematic either way).

[yuyichao]

In another word, if you can get the backtrace without the last ^C it should help =)

[JeffBezanson]

Here are stack traces from every jl_rethrow that happens when I hit ^C in this case:

julia> while true
       end
^C^C^C^C^CWARNING: Force throwing a SIGINT



InterruptException()
anonymous at ./<missing>:0
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_toplevel_eval_flex at /home/jeff/src/julia/src/toplevel.c:569 [inlined]
jl_toplevel_eval at /home/jeff/src/julia/src/toplevel.c:580
jl_toplevel_eval_in_warn at /home/jeff/src/julia/src/builtins.c:576
eval at ./boot.jl:234
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
eval_user_input at ./REPL.jl:62
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
macro expansion at ./REPL.jl:92 [inlined]
#3 at ./event.jl:46
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/src/julia.h:1392 [inlined]
start_task at /home/jeff/src/julia/src/task.c:253
unknown function (ip: 0xffffffffffffffff)



InterruptException()
anonymous at ./<missing>:0
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_toplevel_eval_flex at /home/jeff/src/julia/src/toplevel.c:569 [inlined]
jl_toplevel_eval at /home/jeff/src/julia/src/toplevel.c:580
jl_toplevel_eval_in_warn at /home/jeff/src/julia/src/builtins.c:576
eval at ./boot.jl:234
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
eval_user_input at ./REPL.jl:62
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
macro expansion at ./REPL.jl:92 [inlined]
#3 at ./event.jl:46
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/src/julia.h:1392 [inlined]
start_task at /home/jeff/src/julia/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
^C


InterruptException()
jl_mutex_unlock at /home/jeff/src/julia/src/./julia_threads.h:478 [inlined]
jl_generate_fptr at /home/jeff/src/julia/src/codegen.cpp:969
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:175 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
eval_user_input at ./REPL.jl:58
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
macro expansion at ./REPL.jl:92 [inlined]
#3 at ./event.jl:46
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/src/julia.h:1392 [inlined]
start_task at /home/jeff/src/julia/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
^C


InterruptException()
jl_mutex_unlock at /home/jeff/src/julia/src/./julia_threads.h:478 [inlined]
jl_generate_fptr at /home/jeff/src/julia/src/codegen.cpp:969
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:175 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
eval_user_input at ./REPL.jl:71
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
macro expansion at ./REPL.jl:92 [inlined]
#3 at ./event.jl:46
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:178 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/src/julia.h:1392 [inlined]
start_task at /home/jeff/src/julia/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
^C


InterruptException()
jl_mutex_unlock at /home/jeff/src/julia/src/./julia_threads.h:478 [inlined]
jl_generate_fptr at /home/jeff/src/julia/src/codegen.cpp:969
jl_call_method_internal at /home/jeff/src/julia/src/julia_internal.h:175 [inlined]
jl_apply_generic at /home/jeff/src/julia/src/gf.c:1928
jl_apply at /home/jeff/src/julia/src/julia.h:1392 [inlined]
finish_task at /home/jeff/src/julia/src/task.c:214 [inlined]
start_task at /home/jeff/src/julia/src/task.c:261
unknown function (ip: 0xffffffffffffffff)
fatal: error thrown and no exception handler available.
InterruptException()

It looks like the mutex_unlock from codegen causes re-throwing InterruptException every time?