opal_fifo segfaults on x86_64

[adrianreber]

In my MTT runs on x86_64 opal_fifo sometimes segfaults. It works most of the time but sometimes it is possible to get a segfault like this:

$ LD_LIBRARY_PATH=../../opal/.libs ./opal_fifo
Single thread test. Time: 0 s 23315 us 23 nsec/poppush
Atomics thread finished. Time: 0 s 70590 us 70 nsec/poppush
Atomics thread finished. Time: 1 s 939921 us 1939 nsec/poppush
Atomics thread finished. Time: 2 s 662403 us 2662 nsec/poppush
Atomics thread finished. Time: 3 s 323705 us 3323 nsec/poppush
Atomics thread finished. Time: 3 s 847741 us 3847 nsec/poppush
Atomics thread finished. Time: 3 s 918840 us 3918 nsec/poppush
Atomics thread finished. Time: 3 s 942101 us 3942 nsec/poppush
Atomics thread finished. Time: 3 s 997348 us 3997 nsec/poppush
Atomics thread finished. Time: 4 s 12625 us 4012 nsec/poppush
All threads finished. Thread count: 8 Time: 4 s 25370 us 503 nsec/poppush
[rhlx01:03461] *** Process received signal ***
Segmentation fault (core dumped)

GDB results:

Core was generated by `/home/staff/adrian/openmpi-dev-3613-g6d7ada9/test/class/.libs/lt-opal_fifo'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000401222 in opal_fifo_pop_atomic (fifo=0x7fff548f82e0) at ../../opal/class/opal_fifo.h:127
127             next = (opal_list_item_t *) item->opal_list_next;
[Current thread is 1 (Thread 0x7f2665f86700 (LWP 3513))]
(gdb) bt
#0  0x0000000000401222 in opal_fifo_pop_atomic (fifo=0x7fff548f82e0) at ../../opal/class/opal_fifo.h:127
#1  0x0000000000401607 in thread_test_exhaust (arg=0x7fff548f82e0) at opal_fifo.c:79
#2  0x00007f266a2d260a in start_thread () from /lib64/libpthread.so.0
#3  0x00007f266a00ca4d in clone () from /lib64/libc.so.6

See also here: https://mtt.open-mpi.org/index.php?do_redir=2276

[jsquyres]

@hjelmn @bosilca Can you have a look?

[hjelmn]

I will take a look. What processor is this on?

[jsquyres]

@hjelmn It's x86_64 -- do you need more specific than that?

[hjelmn]

Considering I can't reproduce on sandy bridge, ivy bridge, bulldozer, and a handful of others it will be useful. I may be able to find a system where this occurs.

It would also be good to know what compiler since it could be any number of things. A missing memory barrier, a missing/wrong clobber, etc.

[jsquyres]

@hjelmn Fixed his original comment: he can't reproduce it.

Yeah, it might have to be run in a loop or something...? It seems like a race condition of some kind that only happens sometime.

[adrianreber]

It is a "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz" with 24 HT cores. Fedora 23 with gcc 5.3.1. According to MTT is also happens with the pre-release gcc 6.0 I have installed: gcc-6-20160221

[hjelmn]

gcc 5.3 might be the key. We had a clobber error in the x86 atomic add for forever but it didn't become apparent until gcc 5.0 which had more aggressive optimizations.

[hjelmn]

ok, with gcc 5.3.0 it does cause the opal_fifo test to occasionally segv. investigating.

[hjelmn]

i can't reproduce when builtin sync atomics are enabled. suggests the double-word compare and swap in the amd64 asm is not quite right.

[hjelmn]

ok, looks like it was a bad constraint. testing the fix now.

[hjelmn]

no, looks like an ABA issue. i have a working fix but i want to prove to myself it is the right fix.

[hjelmn]

The issue was complicated but is easily fixed by using an atomic when the head is updated in opal_fifo_push. The atomic shouldn't be necessary but the non-atomic update combined with the way 128-bit values are read (two instructions instead of 1) leads to an ABA issue that puts the fifo in an inconsistent state. See #1470.