Hang in bcast_intra_generic

[rhc54]

This is occurring on master and v3.x, but only in a very specific use-case (which is why it isn't getting picked up in testing). The problem occurs when we are executing a dynamic operation (spawn, connect/accept, etc.) on two nodes where the number of participating procs is identical on each node:

After the child procs successfully start up, everything hangs in bcast_intra_generic:

#0  0x00007f1e8bb46dfd in poll () from /usr/lib64/libc.so.6
#1  0x00007f1e8ce0dff8 in poll_dispatch (base=0xc13dd0, tv=<optimized out>) at poll.c:165
#2  0x00007f1e8ce04ab6 in opal_libevent2022_event_base_loop (base=0xc13dd0, flags=2) at event.c:1630
#3  0x00007f1e8cda0799 in opal_progress () at runtime/opal_progress.c:204
#4  0x00007f1e8d4191f1 in ompi_request_wait_completion (req=0xd08600) at ../ompi/request/request.h:392
#5  0x00007f1e8d41922f in ompi_request_default_wait (req_ptr=0x7ffd57a909c0, status=0x0) at request/req_wait.c:42
#6  0x00007f1e8d4b214a in ompi_coll_base_bcast_intra_generic (buffer=0x7ffd57a90f48, original_count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0, 
    count_by_segment=1, tree=0xd0abe0) at base/coll_base_bcast.c:209
#7  0x00007f1e8d4b282e in ompi_coll_base_bcast_intra_binomial (buffer=0x7ffd57a90f48, count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0, segsize=0)
    at base/coll_base_bcast.c:335
#8  0x00007f1e7b9e3e82 in ompi_coll_tuned_bcast_intra_dec_fixed (buff=0x7ffd57a90f48, count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0)
    at coll_tuned_decision_fixed.c:258
#9  0x00007f1e8d3f9965 in ompi_dpm_connect_accept (comm=0x6025c0 <ompi_mpi_comm_world>, root=0, port_string=0xd08ab0 "2558984193.0:2441757066", send_first=true, newcomm=0x7ffd57a911d0) at dpm/dpm.c:239
#10 0x00007f1e8d3fe9d3 in ompi_dpm_dyn_init () at dpm/dpm.c:1004
#11 0x00007f1e8d41c49a in ompi_mpi_init (argc=0, argv=0x0, requested=0, provided=0x7ffd57a9143c) at runtime/ompi_mpi_init.c:949
#12 0x00007f1e8d45a147 in PMPI_Init (argc=0x0, argv=0x0) at pinit.c:68
#13 0x0000000000400d23 in main (argc=1, argv=0x7ffd57a915c8) at simple_spawn.c:19

The full backtrace of all threads:

(gdb) thread apply all bt

Thread 3 (Thread 0x7f1e8a184700 (LWP 207206)):
#0  0x00007f1e8bb46dfd in poll () from /usr/lib64/libc.so.6
#1  0x00007f1e8ce0dff8 in poll_dispatch (base=0xc2cb40, tv=<optimized out>) at poll.c:165
#2  0x00007f1e8ce04ab6 in opal_libevent2022_event_base_loop (base=0xc2cb40, flags=1) at event.c:1630
#3  0x00007f1e8cda79e9 in progress_engine (obj=0xc172d8) at runtime/opal_progress_threads.c:105
#4  0x00007f1e8be22dc5 in start_thread () from /usr/lib64/libpthread.so.0
#5  0x00007f1e8bb5173d in clone () from /usr/lib64/libc.so.6

Thread 2 (Thread 0x7f1e83fff700 (LWP 207207)):
#0  0x00007f1e8bb51d13 in epoll_wait () from /usr/lib64/libc.so.6
#1  0x00007f1e8ce017f8 in epoll_dispatch (base=0xc5ae10, tv=<optimized out>) at epoll.c:407
#2  0x00007f1e8ce04ab6 in opal_libevent2022_event_base_loop (base=0xc5ae10, flags=1) at event.c:1630
#3  0x00007f1e892ff591 in progress_engine (obj=0xc5ad68) at runtime/pmix_progress_threads.c:110
#4  0x00007f1e8be22dc5 in start_thread () from /usr/lib64/libpthread.so.0
#5  0x00007f1e8bb5173d in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7f1e8d96b740 (LWP 207205)):
#0  0x00007f1e8bb46dfd in poll () from /usr/lib64/libc.so.6
#1  0x00007f1e8ce0dff8 in poll_dispatch (base=0xc13dd0, tv=<optimized out>) at poll.c:165
#2  0x00007f1e8ce04ab6 in opal_libevent2022_event_base_loop (base=0xc13dd0, flags=2) at event.c:1630
#3  0x00007f1e8cda0799 in opal_progress () at runtime/opal_progress.c:204
#4  0x00007f1e8d4191f1 in ompi_request_wait_completion (req=0xd08600) at ../ompi/request/request.h:392
#5  0x00007f1e8d41922f in ompi_request_default_wait (req_ptr=0x7ffd57a909c0, status=0x0) at request/req_wait.c:42
#6  0x00007f1e8d4b214a in ompi_coll_base_bcast_intra_generic (buffer=0x7ffd57a90f48, original_count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0, 
    count_by_segment=1, tree=0xd0abe0) at base/coll_base_bcast.c:209
#7  0x00007f1e8d4b282e in ompi_coll_base_bcast_intra_binomial (buffer=0x7ffd57a90f48, count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0, segsize=0)
    at base/coll_base_bcast.c:335
#8  0x00007f1e7b9e3e82 in ompi_coll_tuned_bcast_intra_dec_fixed (buff=0x7ffd57a90f48, count=1, datatype=0x6021c0 <ompi_mpi_int>, root=0, comm=0x6025c0 <ompi_mpi_comm_world>, module=0xd09ad0)
    at coll_tuned_decision_fixed.c:258
#9  0x00007f1e8d3f9965 in ompi_dpm_connect_accept (comm=0x6025c0 <ompi_mpi_comm_world>, root=0, port_string=0xd08ab0 "2558984193.0:2441757066", send_first=true, newcomm=0x7ffd57a911d0) at dpm/dpm.c:239
#10 0x00007f1e8d3fe9d3 in ompi_dpm_dyn_init () at dpm/dpm.c:1004
#11 0x00007f1e8d41c49a in ompi_mpi_init (argc=0, argv=0x0, requested=0, provided=0x7ffd57a9143c) at runtime/ompi_mpi_init.c:949
#12 0x00007f1e8d45a147 in PMPI_Init (argc=0x0, argv=0x0) at pinit.c:68
#13 0x0000000000400d23 in main (argc=1, argv=0x7ffd57a915c8) at simple_spawn.c:19

Anyone have thoughts?

[rhc54]

Here's the command line: mpirun -n 2 --map-by node ./simple_spawn using the simple_spawn program in orte/test/mpi

[bosilca]

Are we receiving the modex info for the spawned processes ?

[rhc54]

Given that it works whenever the number of procs is uneven on the nodes, it would seem odd that it would be missing modex info causing the problem. Not ruling it out, but seems weird that the modex would fail in only that one case.

[bosilca]

From the MPI perspective there is no such things as spawned processes, as once they are integrated into a communicator they are just processes. So if the MPI_Bcast works for 4 processes spawned by the same mpirun, then it should work for 4 processes from different jobid, as long as we can extract their BTL information from the modex.

Let me try to replicate this.

[rhc54]

ok - thanks!

[bosilca]

I tried to put myself in the same situation. I have a hostfile with 2 names (slots=20), and I use your command line to start the job. I modified the spawner to only have an even number of processes (2, 4, 8), so that they can evenly divide across the 2 nodes. The tests completes every time. I did ensure that the processes are evenly divided across the nodes.

[rhc54]

Hmmm...ok, that's a very interesting data point. Let me dig further. Thx!

[ggouaillardet]

@rhc54 is the app hanging when receiving data from the same node or a remote node ?
which btl(s) are you using ?
what if you ?
```mpirun --mca btl tcp,self ...`

[rhc54]

I honestly don't know where the collective is hanging. I tried with tcp,self, but the behavior is the same. It isn't even a race condition as it happens every single time.

My scenario is, however, a little different than what @bosilca described. The parent has an equal number of procs on the two nodes, but then it spawns 3 children that are unevenly distributed across the nodes. I modified simple_spawn to have it output the hostname of the proc when it announces itself as "starting" so you can see the pattern:

$ mpirun -npernode 2 -mca btl tcp,self ./simple_spawn
[rhc001:87014] starting up!
[rhc002.cluster:194076] starting up!
[rhc001:87015] starting up!
[rhc002.cluster:194077] starting up!
0 completed MPI_Init
Parent [pid 87014] about to spawn!
1 completed MPI_Init
Parent [pid 87015] about to spawn!
2 completed MPI_Init
Parent [pid 194076] about to spawn!
3 completed MPI_Init
Parent [pid 194077] about to spawn!
[rhc002.cluster:194082] starting up!
[rhc002.cluster:194083] starting up!
[rhc001:87020] starting up!
<hang>

Note that the children never get thru MPI_Init, which is expected given that the bt shows everything stuck in the connect/accept bcast. Note also that I then changed simple_spawn so it spawns an even number of procs, thus getting the same number on each node - and it still hung.

The bottom line seems to be with the parents. If the same number of parents are on each node, then we will hang in bcast during connect/accept. If we look at the code referenced in the bt, we see that this is where it hangs:

    /* if we aren't in a comm_spawn, the non-root members won't have
     * the port_string - so let's make sure everyone knows the other
     * side's participants */

    /* bcast the list-length to all processes in the local comm */
    rc = comm->c_coll->coll_bcast(&rportlen, 1, MPI_INT, root, comm,
                                 comm->c_coll->coll_bcast_module);
    if (OMPI_SUCCESS != rc) {
        free(rport);
        goto exit;
    }

So the hanging communicator isn't the intercomm, but rather the original comm of the parents. I don't see how that can have anything to do with modex info as we haven't gotten to the point of exchanging it - we are still in the very earliest stages of the procedure.

[ggouaillardet]

i can reproduce the issue :-)
note i am unable to reproduce it if the node on which mpirun is invoked is not part of the host list
i will dig a bit

[ggouaillardet]

@rhc54 this is the state of the program

• all 3 children are stuck in bcast at dpm.c:239
• parents 1-3 are stuck in bcast at the same location dpm.c:239
• parent 0 is stuck in OPAL_PMIX_EXCHANGE at dpm.c:221

in the children, root is 0 but rank are {1,2,2}
(e.g. there is no child with rank == root)
that explains the hang, but not the root cause (invalid assigned rank)