=test #435 hardening test_singletonByClusterLeadership_withLeaderChange

[ktoso]

It seems the failures are nit really failures of the singleton impl, but hitting slowness on the joining:

00:08:24.039 error: No result within 15s for block at /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:207. Queried 147 times, within 15s. Last error: 
00:08:24.039             try self.ensureNodes(.up, on: second, nodes: fourth.cluster.node, second.cluster.node, third.cluster.node)
00:08:24.039                          ^~~~~~
00:08:24.039 error: Expected sact://fourth:[email protected]:7444 on sact://second:[email protected]:8222 to be seen as: [up], but was [joining]

Adding more logging to test_singletonByClusterLeadership_withLeaderChange.
Also measuring if we're actually dropping messages or not, that's quite good to know (in my testing so far we never dropped (a drop is legal here in some situations)).

Having that said, there will be more testing and optimizing the join speed which would also harden and speed up these.

• Resolves FAILED: test_singletonByClusterLeadership_withLeaderChange #435

[ktoso]

Well, good:

00:09:37.565 Test Case 'ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership_withLeaderChange' started at 2020-06-17 08:01:18.811
00:09:48.893 �[0;32m[8:1:22.2560] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:173 : Nodes up: [sact://first:[email protected]:7111, sact://second:[email protected]:8222, sact://third:[email protected]:9333]�[0;0m
00:09:48.893 �[0;32m[8:1:22.2750] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:182 : All three nodes communicated with singleton�[0;0m
00:09:48.893 �[0;32m[8:1:23.2140] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:196 : Node sact://[email protected]:7111 left cluster...�[0;0m
00:09:48.893 �[0;32m[8:1:23.2150] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:199 : Node sact://[email protected]:7444 joining cluster...�[0;0m
00:09:48.893 �[0;33m[8:1:24.2160] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (1) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:24.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (1) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:25.2160] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (2) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:25.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (2) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:26.2160] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (3) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:26.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (3) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:27.2160] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (4) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:27.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (4) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:28.2160] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (5) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:28.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (5) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:29.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (6) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;33m[8:1:29.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (6) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:48.893 �[0;32m[8:1:30.1370] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:223 : Fourth node joined, will become leader; Members now: [sact://fourth:[email protected]:7444, sact://second:[email protected]:8222, sact://third:[email protected]:9333]�[0;0m
00:09:48.893 �[0;32m[8:1:30.1380] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:228 : Received reply (by /user/$testProbe-0/$messageAdapter) from singleton: Hello-4 Bob-2 (1)!�[0;0m
00:09:49.391 �[0;33m[8:1:30.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:212 :   Sending: Bob-2 (7) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:49.391 �[0;33m[8:1:30.2170] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:216 :   Sending: Charlie-3 (7) -> ActorRef<GreeterSingleton.Message>(/system/singletonProxy-greeter) (it may be terminated/not-re-pointed yet)�[0;0m
00:09:49.391 �[0;32m[8:1:30.3210] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:234 :   No messages were lost! Including 6 more, following the previous delivery.�[0;0m
00:09:49.391 �[0;32m[8:1:30.3220] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:241 : Received reply (by /user/$testProbe-0/$messageAdapter) from singleton: Hello-4 Charlie-3 (1)!�[0;0m
00:09:49.391 �[0;32m[8:1:30.4230] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:247 :   No messages were lost! Including 6 more, following the previous delivery.�[0;0m
00:09:49.391 �[0;32m[8:1:30.4240] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:253 : Singleton re-pointing took: 7s 208ms�[0;0m
00:09:49.391 �[0;32m[8:1:30.4240] /code/Tests/ActorSingletonPluginTests/ActorSingletonPluginClusteredTests.swift:255 : Nodes communicated successfully with singleton on [fourth]�[0;0m
00:09:49.391 Test Case 'ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership_withLeaderChange' passed (11.826 seconds)

Missing removes in DowningClusteredTests.test_many_nonLeaders_shouldPropagateToOtherNodes looking more into it... tbh I think I need to look at the: #672 so will do that, since likely common root cause :)

[ktoso]

Merging since I want to see it a bit more inluding on the mac builds which are a bit slow sometimes