Coll/HAN and Coll/Adapt not default on 5.0.x

[bwbarrett]

We never bumped the priority of the HAN and ADAPT collective components on the 5.0.x branch.

I'm not submitting a PR right now (bumping the priority should be easy) because, at least on EFA, Allgather and Allreduce got considerably slower when using the HAN components. Might be user error, but need to dig more.

[awlauria]

I don't recall if there was any discussion of what priority they should be. @bosilca @janjust @gpaulsen @hppritcha @jsquyres

[jsquyres]

That was definitely the plan: to "preview" han/adapt in the 4.x series and then make it the default to replace "tuned" in 5.x.

[hppritcha]

That was my understanding as well.

[awlauria]

So the priority for both should be higher than tuned.

Did one of han/adapt need to be a higher priority than the other? Or should they be the same priority?

[wckzhang]

IIRC, HAN has more collectives implemented. We are going to do some performance testing on HAN/Adapt/Tuned.

[bwbarrett]

Before we change the priorities, someone with a device other than EFA really needs to run and see if there is benefit closed to that promised in George's paper.

[awlauria]

@bosilca can you post the performance numbers or provide a link for reference?

[bosilca]

Not sure what I'm expected to provide here ?

[awlauria]

1. Are there any known performance data for han/adapt compared to the current ompi defaults?
2. Is there a link to the paper we view?
3. Did you want to open the PR to raise the priorities? You probably have a better understanding of where these priorities should lie.

[bwbarrett]

On EFA, we see essentially no performance difference between today's v5.0.x branch and running with --mca coll_han_priority 100 --mca coll_adapt_priority 100 on the OSU collective benchmarks. In some cases (allreduce in particular), it hurt performance. There are some oddities in EFA's performance on v5.0.x right now, so this may be because of EFA. My ask was that someone with a network that isn't EFA or TCP try the change and verify that the collectives actually do something good. Otherwise, we're advertising improvements that aren't going to be there.

[devreal]

I'm planning to investigate #9062 soon and will also look at the general performance of coll/adapt and coll/han on an IB system soon. Will report back once I have the numbers.