mpi4py: Regression in alltoall with intercommunicators

[dalcinl]

https://github.com/mpi4py/mpi4py-testing/actions/runs/8034284372/job/21945704643

testAlltoallInter (test_util_pkl5.TestMPIWorld.testAlltoallInter) ... python: pml_ob1_comm.c:51: mca_pml_ob1_comm_proc_destruct: Assertion `NULL == proc->frags_cant_match' failed.

BTW, I've submitted #12357 to re-enable multi-proc runs of mpi4py tests. I've just rebased that PR, but now test may fail the same way I'm reporting here.

[jsquyres]

The specific error is:

testAlltoallInter (test_util_pkl5.TestMPIWorld.testAlltoallInter) ... python: pml_ob1_comm.c:51: mca_pml_ob1_comm_proc_destruct: Assertion `NULL == proc->frags_cant_match' failed.
[fv-az979-137:145684] *** Process received signal ***
[fv-az979-137:145684] Signal: Aborted (6)
[fv-az979-137:145684] Signal code:  (-6)

[dalcinl]

The specific error is:

What do you mean? That's exactly what I reported in the description right below the link to the full logs . Am I missing something?

[jsquyres]

That's exactly what I reported in the description right below the link to the full logs . Am I missing something?

Errr... no, sorry. You didn't report the version of Open MPI, and I had to dig into the CI link to find it; I labeled that then copy-and-pasted the specific error out of the logs mostly on auto-pilot (my eyes skipped right over the long line in your description and I missed it).

[hppritcha]

@dalcinl could you remind me which arg to main.py controls the set of tests which are run?

[hppritcha]

okay I figured that out. I'm not able to reproduce this problem running standalone.

[bosilca]

This errors is only possible when out-of-sequence messages are left into a process queue when the communicator is freed or when MPI_Finalize is called. This means there are messages unmatched and out-of-sequence, kind of suggesting some incorrect MPI application.

[dalcinl]

This failing test is doing a series of shot-message Python (pickle-based) comm.alltoall(...) on an intercomm in rapid succession. The test basically reads

comm = make_intercomm(MPI.COMM_WORLD)
size = comm.Get_remote_size()
for smess in messages:
    rmess = comm.alltoall(None)
    assert rmess == [None]*size
    rmess = comm.alltoall([smess]*size
    assert rmess == [smess]*size
    rmess = comm.alltoall(iter([smess]*size))
    assert rmess == [smess]*size
comm.Free()

The algorithm has not change in 15+ years: 1) MPI_Alltoall of counts, and 2) MPI_Alltoallv of the (pickled) byte stream.
https://github.com/mpi4py/mpi4py/blob/master/src/mpi4py/MPI/msgpickle.pxi#L908

This test has been working on MPICH/Intel MPI/Microsoft MPI on Linux/macOS/Windows. I never gave a problem with Open MPI v4.1.x, v5.0.x, and until recently, the main branch. Moreover, the issue is not easy to reproduce, and the test even pass most times in GitHub Actions.

While I understand it is tempting to assume that the problem is on the application (that is, mpi4py) side, IMHO all the evidence points to the backend. A sequence of MPI_Alltoall+MPI_Alltoallv calls occurring simultaneously at all processes should not end up in unmatched messages, unless the MPI implementation is internally messing up things.

I'm indeed freeing the intercommunicator (with Free()) after the many rounds of MPI_Alltoall+MPI_Alltoallv calls. Maybe I should use Disconnect() instead, then rather than an assertion from unmatched messages, we should end up in a deadlock.

[ggouaillardet]

@dalcinl is there a command to run only the test_util_pkl5.TestMPIWorld.testAlltoallInter test?

also, what if you mpirun --mca coll ^han ...

[dalcinl]

@dalcinl is there a command to run only the test_util_pkl5.TestMPIWorld.testAlltoallInter test?

The main entry point to run everything is test/main.py, but every test/test_*.py script can be run individually. In turn, there are a few different ways to run a specific test (all following Python's standard unittest infrastructure), but the most obvious/easy one probably is:

mpiexec -n 5 python test/main.py -v -k test_util_pkl5.TestMPIWorld.testAlltoallInter

PS: Run python test/main.py --help for all the options...

also, what if you mpirun --mca coll ^han ...

I'll try locally, but given that I cannot reproduce that way, I'm not sure such a try will be really relevant.

[dalcinl]

OK, now I can reproduce. I have a machine with 128 virtual cores.

If I run with 127 mpi4py processes it seems to work most of the times:

mpiexec -n 127 python test/main.py -v -k test_util_pkl5.TestMPIWorld.testAlltoallInter

but as soon as I ask for 128 MPI processes, then the assertion triggers:

mpiexec -n 127 python test/main.py -v -k test_util_pkl5.TestMPIWorld.testAlltoallInter
...
[31@kw61149] numpy 1.26.4 (/home/dalcinl/.local/lib/python3.12/site-packages/numpy)
[31@kw61149] MPI 3.1 (Open MPI 5.1.0)
[31@kw61149] mpi4py 4.0.0.dev0 (/home/dalcinl/Devel/mpi4py/build/lib.linux-x86_64-cpython-312/mpi4py)
testAlltoallInter (test_util_pkl5.TestMPIWorld.testAlltoallInter) ... python: ../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_comm.c:51: mca_pml_ob1_comm_proc_destruct: Assertion `NULL == proc->frags_cant_match' failed.
[kw61149:3686873] *** Process received signal ***
[kw61149:3686873] Signal: Aborted (6)
[kw61149:3686873] Signal code:  (-6)
[kw61149:3686873] [ 0] /lib64/libc.so.6(+0x3e9a0)[0x7f7eca2a39a0]
[kw61149:3686873] [ 1] /lib64/libc.so.6(+0x90834)[0x7f7eca2f5834]
[kw61149:3686873] [ 2] /lib64/libc.so.6(raise+0x1e)[0x7f7eca2a38ee]
[kw61149:3686873] [ 3] /lib64/libc.so.6(abort+0xdf)[0x7f7eca28b8ff]
[kw61149:3686873] [ 4] /lib64/libc.so.6(+0x2681b)[0x7f7eca28b81b]
[kw61149:3686873] [ 5] /lib64/libc.so.6(+0x36c57)[0x7f7eca29bc57]
[kw61149:3686873] [ 6] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x377734)[0x7f7ebb177734]
[kw61149:3686873] [ 7] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x3771b5)[0x7f7ebb1771b5]
[kw61149:3686873] [ 8] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x377ce5)[0x7f7ebb177ce5]
[kw61149:3686873] [ 9] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x3736a7)[0x7f7ebb1736a7]
[kw61149:3686873] [10] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(mca_pml_ob1_del_comm+0xe1)[0x7f7ebb17584d]
[kw61149:3686873] [11] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x6345b)[0x7f7ebae6345b]
[kw61149:3686873] [12] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(+0x6440c)[0x7f7ebae6440c]
[kw61149:3686873] [13] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(ompi_comm_free+0x48e)[0x7f7ebae6a4fc]
[kw61149:3686873] [14] /home/devel/mpi/openmpi/main/lib/libmpi.so.0(PMPI_Comm_free+0x11d)[0x7f7ebaedf8cc]
[kw61149:3686873] [15] /home/dalcinl/Devel/mpi4py/build/lib.linux-x86_64-cpython-312/mpi4py/MPI.cpython-312-x86_64-linux-gnu.so(+0x14ae82)[0x7f7ebb54ae82]
[kw61149:3686873] [16] /home/dalcinl/Devel/mpi4py/build/lib.linux-x86_64-cpython-312/mpi4py/MPI.cpython-312-x86_64-linux-gnu.so(+0x14ae29)[0x7f7ebb54ae29]
[kw61149:3686873] [17] /home/dalcinl/Devel/mpi4py/build/lib.linux-x86_64-cpython-312/mpi4py/MPI.cpython-312-x86_64-linux-gnu.so(+0x338511)[0x7f7ebb738511]
[kw61149:3686873] [18] /lib64/libpython3.12.so.1.0(PyObject_Vectorcall+0x5c)[0x7f7ec9e0ae9c]
[kw61149:3686873] [19] /lib64/libpython3.12.so.1.0(+0x10fb3a)[0x7f7ec9d0fb3a]
[kw61149:3686873] [20] /lib64/libpython3.12.so.1.0(+0x234b7d)[0x7f7ec9e34b7d]
[kw61149:3686873] [21] /lib64/libpython3.12.so.1.0(+0x1105fb)[0x7f7ec9d105fb]
[kw61149:3686873] [22] /lib64/libpython3.12.so.1.0(_PyObject_FastCallDictTstate+0xa6)[0x7f7ec9df9b66]
[kw61149:3686873] [23] /lib64/libpython3.12.so.1.0(_PyObject_Call_Prepend+0x73)[0x7f7ec9e1e4d3]
[kw61149:3686873] [24] /lib64/libpython3.12.so.1.0(+0x2cb485)[0x7f7ec9ecb485]
[kw61149:3686873] [25] /lib64/libpython3.12.so.1.0(_PyObject_MakeTpCall+0x76)[0x7f7ec9df7466]
[kw61149:3686873] [26] /lib64/libpython3.12.so.1.0(+0x10fb3a)[0x7f7ec9d0fb3a]
[kw61149:3686873] [27] /lib64/libpython3.12.so.1.0(+0x234b7d)[0x7f7ec9e34b7d]
[kw61149:3686873] [28] /lib64/libpython3.12.so.1.0(+0x1105fb)[0x7f7ec9d105fb]
[kw61149:3686873] [29] /lib64/libpython3.12.so.1.0(_PyObject_FastCallDictTstate+0xa6)[0x7f7ec9df9b66]
[kw61149:3686873] *** End of error message ***
--------------------------------------------------------------------------
    This help section is empty because PRRTE was built without Sphinx.
--------------------------------------------------------------------------

The --mca ^han option is not working on my side, but most likely because the option syntax is incorrect?

mpiexec -n 2 --mca ^han python test/main.py -v -k test_util_pkl5.TestMPIWorld.testAlltoallInter
--------------------------------------------------------------------------
    This help section is empty because PRRTE was built without Sphinx.
--------------------------------------------------------------------------