Skip to content

Conversation

@lrbison
Copy link
Contributor

@lrbison lrbison commented Jan 24, 2024

PR for discussion on how to add smsc endpoints to HAN for local memory exchange.

Motivation: I have a working POC using XPMEM and HAN to increase alltoall exchange speed, but I would prefer to cache the creation of the XPMEM endpoints rather than creating them new each time in the collective.

Questions:

  • Where to store the endpoints, and how to properly initialize and deconstruct them.
  • What is the difference between mca_coll_han_comm_create_new and mca_coll_han_comm_create?

Signed-off-by: Luke Robison <[email protected]>
@lrbison
Copy link
Contributor Author

lrbison commented Jan 29, 2024

Motivating this change is a branch I'm working on to add alltoall (and soon alltoallv) algorithm to HAN.

My testing shows that:

  • SMSC End point creation is to slow to be done on-the-fly, so we need a cache somewhere.
  • XPMEM SMSC module has the lowest latency, beating out CMA and KNEM, even for copy_to/from, but especially using mapping.

Here are the results I have so far:

image

@lrbison lrbison closed this Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant