Reduce locking or lockless Associations

[ktoso]

In today's impl we are correct with associations however we require locks around them.

• we need to allow concurrently looking up Association when sending a ref by hitting the ClusterShell
• we need to lock while we perform sends so we're not accidentally enqueueing into the queue while it should already be put on the channel

One way around this is to always go through the queue, though then the polling from it has to be done somewhere (RS style signaling we have elements, hello :P)

There's multiple phases we could push through and they have tradeoffs.

We could try to reduce the first contention point (let's first measure it -- benchmark with talking to many remotes, should show this (from many other actors).

The ClusterShell one seems like it may be a candidate to RWLock and get away with it. A concurrent dict would be nice there; and then inside the Assoc we could do the Queue dance...

Just some thoughts, we'd have to look and see

[ktoso]

We did the long awaited optimization that the remote personality can cache the association, I think this solves a lot of this topic: 049487f