pml/ucx: Propagate MPI serialized thread mode #12579
Conversation
The UCX PML needs to differentiate between mpi serialized and single thread mode according to MPI standard. Using UCX with multiple threads, even when providing mutual exclusion, is not completely equivalent to only using UCX with one same thread. Signed-off-by: Thomas Vegas <[email protected]> (cherry picked from commit b33b61a)
|
Shall we also backport it to v4.1.x? |
| case MPI_THREAD_FUNNELED: | ||
| case MPI_THREAD_SINGLE: | ||
| default: | ||
| return UCS_THREAD_MODE_SINGLE; |
There was a problem hiding this comment.
default should be multi, or at least produce a warning
There was a problem hiding this comment.
Good catch. This should indeed be moved in UCX common (opal/mca/common/ucx/common_ucx_wpool.c seems like a good place) and reused in all the UCX (and maybe UCC) initializations.
For MPI:
ompi/mca/pml/ucx/pml_ucx.compi/mca/coll/ucc/coll_ucc_module.c
For OSHMEM:
oshmem/mca/spml/ucx/spml_ucx.coshmem/mca/spml/ucx/spml_ucx_component.coshmem/mca/scoll/ucc/scoll_ucc_module.c
| return OMPI_SUCCESS; | ||
| } | ||
|
|
||
| static ucs_thread_mode_t mca_pml_ucx_thread_mode(int ompi_mode) |
There was a problem hiding this comment.
move the function on opal/mca/common/ucx and reuse it also for osc/ucx
There was a problem hiding this comment.
seems osc/ucx is in v4.1.x (thanks for noting @brminich), will add to patch when backporting
|
Based on @yosefe and @bosilca comments, I will close this PR, address them with new PR on I will move added function with added warning to Suggested files:
Please let me know otherwise. |
|
superseded by #12608 |
4 participants
cherry-pick #12577
What
The UCX PML needs to differentiate between mpi serialized and single thread mode according to MPI standard.
Why
Using UCX with multiple threads, even when providing mutual exclusion, is not completely equivalent to only using UCX with one same thread.
To guarantee ordering and avoid corruption, some UCX transports need the cpu to flush stores to the devices, before another thread can progress the worker. For latency reasons, this needed flush is only performed in non single thread mode.