v2.1.0: Fix OOS issues in openib BTL with single-threaded scenarios

[jsquyres]

Per #2067, the openib BTL has some cases where out-of-sequence messages can occur, even in single-threaded scenarios. This affects master, v2.0.x, v2.1.x. It assumedly affects v1.10.x, but I don't know if anyone cares.

(note that this issue specifically does not address the OOS issues in multi-threaded scenarios -- that is a much larger issue, and is covered by #2159)

@larrystevenwise @bharatpotnuri Can you guys have a look at this? See #2067 for reference.

[larrystevenwise]

@jsquyres - looking at mca_btl_openib_sendi(), it returns OPAL_ERR_RESOURCE_BUSY if various resources aren't available. I don't see it queuing the message anywhere for later retry that could result in OOS. Am I all wet?

[jsquyres]

@hjelmn Can you comment? I haven't looked in the openib BTL in years.

[hjelmn]

I think this is largely due to the send coalescing code. I asked @thananon to verify.

[jsquyres]

@larrystevenwise @bharatpotnuri See the thread starting around here #2067 (comment) for some details.

[jsquyres]

@larrystevenwise @bharatpotnuri Does Chelsio intend to look at this? At this point, this issue will need to be re-milestoned to v3.0.0.

[thananon]

Ok, so I measured OOS message in current master with MPI_THREAD_SINGLE/ openib btl. I would say this problem is not yet solved. Unfortunately my work is taking hit by this problem.

Number of messages	OOS count	Percentage
256000	54146	21%
512000	62007	12%
1024000	64378	8%

[larrystevenwise]

We'll look at it at some point, but we're swamped right now.

[hppritcha]

@larrystevenwise @bharatpotnuri any chance someone will look at this soon? otherwise we'll move to future milestone.