OSHMEM/spml/ucx: Race condition in the multi-threaded Private contexts situation #7029
Description
When the context is created as private underlying UCX worker is created in the unprotected mode [link] (no locking is used to ensure the thread-safety of the worker).
According to the OSHMEM spec, the private context is not intended for the shared use and is not expected to be progressed by any thread other than a creator (owning thread).
In shmem_ctx_quiet(), the calling thread is mainly progressing the UCX worker that corresponds to the context passed as the argument [link]. However once in a while, it also calls opal_progress() [link].
The opal_progress() is a global function that, in particular, performs the progress on all of the context workers (including private/unprotected ones) [link] that in parallel may be progressed by an owning thread causing a race condition.
Solution (based on the discussion with @manjugv and @janjust):
The private contexts should not be globally exposed and should not be added to mca_spml_ucx.active_array list.
FYI: @jladd-mlnx, @yosefe, @brminich, @hoopoepg