Repeated calls to MPI_Cart_create segfaults or hangs when first argument is a cart comm

[wrs20]

Fails in OpenMPI 4.0.0 and 3.1.3, passes in OpenMPI 2.1.1 (and MPICH).
OpenMPI installed from 4.0.0 source tar. Ubuntu 18.04.2.

Source:

// cart_break.c
#include <mpi.h>
int main(int argc, char * argv[]){
   
    MPI_Init(&argc, &argv);
    MPI_Comm parent0, parent1, child0, child1;

    int ndims2[3] = {2, 1, 1};
    int ndims1[3] = {1, 1, 1};
    int periods[3] = {1, 1, 1};
    
    // parent 0 (works)
    if (MPI_Cart_create(MPI_COMM_WORLD, 3, ndims2, periods, 0, &parent0) != MPI_SUCCESS) { return -1; }

    // child0 from parent0 (works)
    if(MPI_Cart_create(parent0, 3, ndims1, periods, 0, &child0) != MPI_SUCCESS) { return -1; }


    // parent 1 (works)
    if(MPI_Cart_create(MPI_COMM_WORLD, 3, ndims2, periods, 0, &parent1) != MPI_SUCCESS){ return -1; }

    // child1 from parent1 (hangs in mpi4py, segfaults in c) passes if parent1 is replaced with either parent0 or MPI_COMM_WORLD
    if(MPI_Cart_create(parent1, 3, ndims1, periods, 0, &child1) != MPI_SUCCESS) {return -1;}
    

    // cleanup
    if (child0 != MPI_COMM_NULL){ MPI_Comm_free(&child0); }
    if (child1 != MPI_COMM_NULL){ MPI_Comm_free(&child1); }
    if (parent0 != MPI_COMM_NULL){ MPI_Comm_free(&parent0); }
    if (parent1 != MPI_COMM_NULL){ MPI_Comm_free(&parent1); }


    MPI_Finalize();
    return 0;
}

To reproduce:

shell$ mpicc cart_break.c
shell$ mpirun -n 2 ./a.out

Traceback:

[pypc-dm-07:10294] *** Process received signal ***
[pypc-dm-07:10294] Signal: Segmentation fault (11)
[pypc-dm-07:10294] Signal code: Address not mapped (1)
[pypc-dm-07:10294] Failing at address: 0x8
[pypc-dm-07:10294] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f34b01f6f20]
[pypc-dm-07:10294] [ 1] /home/wrs20/opt/openmpi-4.0.0/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_match+0x995)[0x7f349f3cf1f5]
[pypc-dm-07:10294] [ 2] /home/wrs20/opt/openmpi-4.0.0/lib/openmpi/mca_btl_vader.so(mca_btl_vader_poll_handle_frag+0x8f)[0x7f349eb0439f]
[pypc-dm-07:10294] [ 3] /home/wrs20/opt/openmpi-4.0.0/lib/openmpi/mca_btl_vader.so(+0x46c7)[0x7f349eb046c7]
[pypc-dm-07:10294] [ 4] /home/wrs20/opt/openmpi-4.0.0/lib/libopen-pal.so.40(opal_progress+0x2c)[0x7f34afc1cc2c]
[pypc-dm-07:10294] [ 5] /home/wrs20/opt/openmpi-4.0.0/lib/libmpi.so.40(ompi_comm_nextcid+0x105)[0x7f34b05deb25]
[pypc-dm-07:10294] [ 6] /home/wrs20/opt/openmpi-4.0.0/lib/libmpi.so.40(ompi_comm_enable+0x39)[0x7f34b05dc0f9]
[pypc-dm-07:10294] [ 7] /home/wrs20/opt/openmpi-4.0.0/lib/libmpi.so.40(mca_topo_base_cart_create+0x1c4)[0x7f34b0683a54]
[pypc-dm-07:10294] [ 8] /home/wrs20/opt/openmpi-4.0.0/lib/libmpi.so.40(MPI_Cart_create+0x25f)[0x7f34b061299f]
[pypc-dm-07:10294] [ 9] a.out(+0xa68)[0x56198c604a68]
[pypc-dm-07:10294] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f34b01d9b97]
[pypc-dm-07:10294] [11] a.out(+0x84a)[0x56198c60484a]
[pypc-dm-07:10294] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

Many Thanks,
Will

[bosilca]

@wrs20 I cannot replicate your bug in the upcoming 4.1 nor in master, but the symptoms reminded me of a bug fixed not long ago in other branches (fb17115). Can you try with a nightly version or with a 4.x prerelease ?

[ggouaillardet]

@bosilca same thing here, but I have a different issue with v3.1.x. I am now refreshing my install and will double check again.

[ggouaillardet]

@bosilca never mind, I messed up in my test. the latest v3.1.x is working just fine

[wrs20]

@bosilca, @ggouaillardet I built the master-201903270305-4e5cacc snapshot with --enable-debug --enable-mem-debug --enable-mpi1-compatibility and it works fine. So this looks resolved in later versions?

To see if I could find anything useful, I then rebuilt the 4.0.0 release with the above flags and it still segfaults.

Running through gdb gives:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff536b700 (LWP 17801)]
[New Thread 0x7fffee989700 (LWP 17802)]
[pypc-dm-07:17793] ompi_comm_peer_lookup: invalid peer index (1)

Thread 1 "a.out" received signal SIGSEGV, Segmentation fault.
0x00007fffe6d8a2a8 in mca_pml_ob1_peer_lookup (comm=0x5555558b90e0, rank=1) at pml_ob1_comm.h:78
78                  OBJ_RETAIN(proc->ompi_proc);
(gdb) where
#0  0x00007fffe6d8a2a8 in mca_pml_ob1_peer_lookup (comm=0x5555558b90e0, rank=1) at pml_ob1_comm.h:78
#1  0x00007fffe6d8bb26 in mca_pml_ob1_recv_frag_callback_match (btl=0x7fffe68b61c0 <mca_btl_vader>, tag=65 'A', des=0x7fffffffcf50, cbdata=0x0) at pml_ob1_recvfrag.c:362
#2  0x00007fffe66ad658 in mca_btl_vader_poll_handle_frag (hdr=0x7fffe4790948, endpoint=0x5555558ab8a0) at btl_vader_component.c:627
#3  0x00007fffe66ad774 in mca_btl_vader_poll_fifo () at btl_vader_component.c:651
#4  0x00007fffe66ada38 in mca_btl_vader_component_progress () at btl_vader_component.c:736
#5  0x00007ffff6fb4190 in opal_progress () at runtime/opal_progress.c:231
#6  0x00007ffff7a2e345 in ompi_request_wait_completion (req=0x5555558b6658) at ../ompi/request/request.h:415
#7  0x00007ffff7a2eb96 in ompi_comm_nextcid (newcomm=0x5555558bce10, comm=0x5555558bae60, bridgecomm=0x0, arg0=0x0, arg1=0x0, send_first=false, mode=32) at communicator/comm_cid.c:293
#8  0x00007ffff7a2d754 in ompi_comm_enable (old_comm=0x5555558bae60, new_comm=0x5555558bce10, new_rank=0, num_procs=1, topo_procs=0x5555558bcdf0) at communicator/comm.c:1888
#9  0x00007ffff7b3518f in mca_topo_base_cart_create (topo=0x5555558bcc90, old_comm=0x5555558bae60, ndims=3, dims=0x7fffffffd3a0, periods=0x7fffffffd3ac, reorder=false, comm_topo=0x7fffffffd388) at base/topo_base_cart_create.c:163
#10 0x00007ffff7a7c3f4 in PMPI_Cart_create (old_comm=0x5555558bae60, ndims=3, dims=0x7fffffffd3a0, periods=0x7fffffffd3ac, reorder=0, comm_cart=0x7fffffffd388) at pcart_create.c:104
#11 0x0000555555554a68 in main ()
(gdb) p proc
$1 = (mca_pml_ob1_comm_proc_t *) 0x5555558bd5f0
(gdb) p proc->ompi_proc
$2 = (struct ompi_proc_t *) 0x0

I can investigate further if it would be useful.

[ggouaillardet]

@wrs20 the issue has been fixed in the release branches, so no investigation is needed.

Meanwhile, you can give 4.0.1rc3 a try

[wrs20]

@ggouaillardet 4.0.1rc3 built with the above flags seems to work. Thanks!

[jsquyres]

[awlauria]

Re-closing since #6522 was merged.