=singleton #346 stabilize test_singletonByClusterLeadership_withLeaderChange

[ktoso]

Motivation:

Ended up stabilizing the test by lowering the assumptions made by the test.

I had some quite progressed up numbers code really didn't like some bits of it so stashed it for now... Will revive it though so we can implement the "oldest" properly. Will do this in separate mini PR.

I've not seen this hardened test fail (nor drop a message), so want to see it on CI as well.
Continuing with other work around this area.

Modifications:

• harden the joining ensuring status a bit in the test
• technically loss is possible here, but I've not seen it with those changes

I'm following up with further cluster lifecycle missing features fixing:

• shutdown when downed Node upon being .down should initiate shutdown #55
• remove when all know Move nodes in membership to removed when ready to do so #52

Result:

• Resolves FAILED: ActorSingletonPluginTests.test_singletonByClusterLeadership #346

[ktoso]

Logs don't indicate we dropped any message either...

22:01:41 Test Case 'ActorSingletonPluginTests.test_singletonByClusterLeadership_withLeaderChange' started at 2020-01-17 13:01:41.018
22:01:47 Test Case 'ActorSingletonPluginTests.test_singletonByClusterLeadership_withLeaderChange' passed (6.186 seconds)

If it would we have

 pprint("lost msg @ node 2")

And even if it did, technically yes this "could happen" as we don't guarantee delivery -- we can later on provide it as an additional layer though.

Please have a look @yim-lee .

Having that said, I'm still working on the "oldest" as well as other cluster hardening.

[yim-lee]

<3

[ktoso]

Thanks for review :)