Performance of freelists and other common OPAL classes with OPAL_ENABLE_MULTI_THREADS==1

[jsquyres]

Per the thread started here (http://www.open-mpi.org/community/lists/devel/2014/10/16130.php), the performance of critical OPAL classes has gotten more expensive on master, now that we have set OPAL_ENABLE_MULTI_THREADS to 1 (this was not unexpected).

However, we now have a definite metric for it -- the PSM message rate falls quite a bit (sidenote: this may also be what inspired Mellanox to write the yalla PML; they may have noticed their message rate was artificially too low in the MXM MTL).

So -- let's figure out how to recover the performance (when possible). Obvious options:

1. Only enable the more expensive forms of the classes when THREAD_MULTIPLE and/or async progress threads are being used. I mention this option for completeness; it does not seem like the right solution because it somewhat defeats the point of setting OPAL_ENABLE_MULTI_THREADS to 1 by default.
2. Have multiple flavors of OPAL classes that are performance-critical (e.g., freelists): thread-safe and non-thread-safe. Code can then choose which flavor to use, depending on the specific situation (e.g., whether it's data structures that are potentially shared among multiple threads or data structures that are guaranteed to always be used by a single thread).

Adding those who will likely care / have opinions here: @bosilca @hjelmn @miked-mellanox @jladd-mlnx @ggouaillardet

[mike-dubman]

Adding @yosefe
I think number 2 from your list is a way to go.

[yosefe]

Yes, 2 is what you really want. The fact that an application can have multiple threads does not necessarily mean that every data structure used has to be thread safe all the time.

[hjelmn]

Ok, I think we should crush opal_free_list_t and ompi_free_list_t into a single type at the same time. We could then provide both atomic and non-atomic flavors.

[hjelmn]

Would also allow us to get rid of the likes of ompi_free_list_init_new_ex ;)

[jladd-mlnx]

+1 for #2.

On Sun, Nov 2, 2014 at 3:44 AM, Mike Dubman notifications@github.com
wrote:

• @yosefe https://github.com/yosefe I think number 2 from your list
  is a way to go.

—
Reply to this email directly or view it on GitHub
#258 (comment).

[rhc54]

I'm puzzled - we already follow the 2nd approach in some classes, as we decided at an earlier OMPI developer meeting. Why are we revisiting the question now? Isn't the issue more that (a) we didn't extend the method to cover other required classes, and (b) people didn't follow thru on using the different class distinctions?

[jsquyres]

@rhc54 Yes, my option 2 was inspired by this decision from a prior meeting.

I think the case is exactly what you said: both (a) and (b).

@hjelmn Also raises a good point -- it would be good to squash down OPAL and OMPI free lists, and git rid of those "new" methods.

@hjelmn Could you make a proposal for a new set of combined OPAL / OMPI free lists that includes thread-safe and not-thread-safe versions? (I think the convention we agreed to before was to add "_mt" and "_MT" to the versions that support Multiple Threads)