Is there a reason why ```opal_atomic_mb``` is always issued?

[joerowell]

I've recently been profiling some OpenMPI-based code and I've noticed that every call to a test function (e.g. MPI_Testsome) executes a memory fence via opal_atomic_mb.

I understand why you might want this in multi-threaded code (either via progress threads or multi-threaded user code), or even in light of the new partitioned collectives in MPI-4.0.

However, OpenMPI issues this instruction even if MPI is initialised with MPI_THREAD_SINGLE or if progress threads are disabled (i.e. OPAL_ENABLE_PROGRESS_THREADS == 0).

Is there a reason for this?

Thanks in advance!

[devreal]

I think that is a valid question and I don't see a good reason for it as the user code won't have modified any internal data that other threads in MPI will need to see and all communicated data is already visible. I didn't write that code though. Maybe someone who wrote this code remembers why the memory barrier is there? (git attributes that line to @rhc54 that might be a fluke)

[joerowell]

Thanks for the quick reply! If there's no need for it, I'm happy to submit a PR that's more "conditional", but I'll wait for others to opine on why it's there to begin with.

[bosilca]

I concur, in single threaded scenarios when the progress thread is disabled this should be a no-op. It seems we just missed it in mca/threads/thread_usage.h where we have similar constructs for all the other atomic/thread-safe operations.

[devreal]

Even in thread-parallel executions it is not clear to me why that memory barrier is needed. What data structures have been modified before the call to MPI_Test* that must be visible to other threads?

[bosilca]

The requests if they were completed onto another thread.

[devreal]

Does that require a full memory barrier though? I can see that there should be an acquire-fence between checking REQUEST_COMPLETE and reading the status, to make sure that there is no reordering between the two. The request array itself is never accessed by another thread and the requests are not modified by the calling thread after the last call into MPI (which must already fence in some way to make sure other threads see consistent request objects). So the full barrier in the beginning is still not clear to me.

[bosilca]

This memory barrier was there from the beginning of the OMPI project, introduced in 2004 by Tim Woodall (2ce7ca7). But @devreal is right, we need a acquire read between noticing completion and accessing the status instead of this generic memory barrier.

[devreal]

I'll work on a patch 👍