FAILED: ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership

[yim-lee]

09:12:27 /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:22: error: ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership : failed - 
09:12:27         try shouldNotThrow {
09:12:27             ^~~~~~~~~~~~~~~
09:12:27 error: Unexpected throw captured
09:12:27             try replyProbe3.expectMessage("Hello-1 Charlie!")
09:12:27                             ^~~~~~~~~~~~~~
09:12:27 error: Did not receive expected [Hello-1 Charlie!]:String within [3s], error: noMessagesInQueue
09:12:27 <EXPR>:0: error: ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership : threw error "unexpectedErrorWasThrown"

A result of changes in #458

[yim-lee]

first node (leader, spawns singleton):

09:12:27 Captured log [first][2020-02-21 17:12:24.1570] [ActorSingletonProxy.swift:112][/system/singletonProxy-greeter] [debug] Node sact://first@localhost:7111 taking over singleton greeter
09:12:27 Captured log [first][2020-02-21 17:12:24.1590] [ActorShell+Children.swift:345][/system/singletonManager-greeter] [debug] Spawning [DistributedActors.Behavior<ActorSingletonPluginTests.GreeterSingleton.Message>.setup((Function))], on path: [/system/singletonManager-greeter/greeter]

third node:

09:12:27 Captured log [third][2020-02-21 17:12:24.1570] [ActorSingletonProxy.swift:170][/system/singletonProxy-greeter] [debug] Updating ref from [nil] to [Optional(ActorRef<GreeterSingleton.Message>(sact://first@localhost:7111/system/singletonProxy-greeter))], flushing 1 messages
09:12:27 Captured log [third][2020-02-21 17:12:24.1570] [ActorSingletonProxy.swift:179][/system/singletonProxy-greeter] [debug] Flushing greet(name: "Charlie", _replyTo: ActorRef<String>(/user/$testProbe-0/$messageAdapter)), to ActorRef<GreeterSingleton.Message>(sact://first@localhost:7111/system/singletonProxy-greeter)

This is a race of other nodes seeing the leadership before leader node can spawn the singleton.

Will try to reduce flakiness in tests by bringing up ActorSingletonProxy actors (which spawn manager based on cluster events) before setting up cluster. (2c23474)

[ktoso]

Tried to make it stable but struggled a bit, going to revisit soon.