FAILED: ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing

[ktoso]

May need a lock there (likely noted there a TODO even?)...

Test Case 'ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing' started at 2019-08-21 07:57:02.793
Assertion failed: Channel should always be present after the initial initialization.: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 181
Current stack trace:
0    libswiftCore.so                    0x00007fba5472d7a0 _swift_stdlib_reportFatalErrorInFile + 115
1    libswiftCore.so                    0x00007fba546689cc <unavailable> + 3463628
2    libswiftCore.so                    0x00007fba54668abe <unavailable> + 3463870
3    libswiftCore.so                    0x00007fba5446431a <unavailable> + 1348378
4    libswiftCore.so                    0x00007fba5463cab2 <unavailable> + 3283634
5    libswiftCore.so                    0x00007fba544635e9 <unavailable> + 1345001
6    DistributedActorsPackageTests.xctest 0x0000561aa8d8a958 <unavailable> + 3987800
7    DistributedActorsPackageTests.xctest 0x0000561aa8d8bb51 <unavailable> + 3992401
8    DistributedActorsPackageTests.xctest 0x0000561aa8d5f6ea <unavailable> + 3811050
9    DistributedActorsPackageTests.xctest 0x0000561aa8d5dee1 <unavailable> + 3804897
10   DistributedActorsPackageTests.xctest 0x0000561aa8d62dff <unavailable> + 3825151
11   DistributedActorsPackageTests.xctest 0x0000561aa8d7cc9b <unavailable> + 3931291
12   DistributedActorsPackageTests.xctest 0x0000561aa8ce933e <unavailable> + 3326782
13   DistributedActorsPackageTests.xctest 0x0000561aa8f6f25f <unavailable> + 5972575
14   DistributedActorsPackageTests.xctest 0x0000561aa8f6ddae <unavailable> + 5967278
15   DistributedActorsPackageTests.xctest 0x0000561aa8f6d9e6 <unavailable> + 5966310
16   DistributedActorsPackageTests.xctest 0x0000561aa8cb0f66 <unavailable> + 3096422
17   DistributedActorsPackageTests.xctest 0x0000561aa8e7c538 <unavailable> + 4977976
18   DistributedActorsPackageTests.xctest 0x0000561aa8e7c936 <unavailable> + 4978998
19   DistributedActorsPackageTests.xctest 0x0000561aa8e7abd1 <unavailable> + 4971473
20   DistributedActorsPackageTests.xctest 0x0000561aa8e7a831 <unavailable> + 4970545
21   DistributedActorsPackageTests.xctest 0x0000561aa8e7aeb9 <unavailable> + 4972217
22   DistributedActorsPackageTests.xctest 0x0000561aa8abc554 <unavailable> + 1045844
23   DistributedActorsPackageTests.xctest 0x0000561aa8e8168e <unavailable> + 4998798
24   DistributedActorsPackageTests.xctest 0x0000561aa8e8aef9 <unavailable> + 5037817
25   DistributedActorsPackageTests.xctest 0x0000561aa8ce2cac <unavailable> + 3300524
26   DistributedActorsPackageTests.xctest 0x0000561aa8e6ef71 <unavailable> + 4923249
27   DistributedActorsPackageTests.xctest 0x0000561aa8ce30dc <unavailable> + 3301596
28   DistributedActorsPackageTests.xctest 0x0000561aa8e6f0c1 <unavailable> + 4923585
29   DistributedActorsPackageTests.xctest 0x0000561aa8e6e9c1 <unavailable> + 4921793
30   DistributedActorsPackageTests.xctest 0x0000561aa8e6eb31 <unavailable> + 4922161
31   DistributedActorsPackageTests.xctest 0x0000561aa8f837b7 <unavailable> + 6055863
32   DistributedActorsPackageTests.xctest 0x0000561aa8f83849 <unavailable> + 6056009
33   DistributedActorsPackageTests.xctest 0x0000561aa8f846d5 <unavailable> + 6059733
34   DistributedActorsPackageTests.xctest 0x0000561aa8f84729 <unavailable> + 6059817
35   libpthread.so.0                    0x00007fba541036db <unavailable> + 30427
36   libc.so.6                          0x00007fba525ad850 clone + 63
/code/Tests/DistributedActorsTests/Cluster/AssociationClusteredTests.swift:137: error: ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing : failed - 
        _ = try p7337.expectMessage()
                     ^~~~~~~~~~~~~~
error: Did not receive message of type [HandshakeResult] within [3s], error: noMessagesInQueue

[ktoso]

Again #25

Will have a look at it shortly, need to stabilize tests on the new CI

10:38:19 Test Case 'ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing' started at 2019-08-22 01:38:19.462
10:38:19 Assertion failed: Channel should always be present after the initial initialization.: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 181
10:38:19 Current stack trace:
10:38:19 0    libswiftCore.so                    0x00007fe6b26687a0 _swift_stdlib_reportFatalErrorInFile + 115
10:38:19 1    libswiftCore.so                    0x00007fe6b25a39cc <unavailable> + 3463628
10:38:19 2    libswiftCore.so                    0x00007fe6b25a3abe <unavailable> + 3463870
10:38:19 3    libswiftCore.so                    0x00007fe6b239f31a <unavailable> + 1348378
10:38:19 4    libswiftCore.so                    0x00007fe6b2577ab2 <unavailable> + 3283634
10:38:19 5    libswiftCore.so                    0x00007fe6b239e5e9 <unavailable> + 1345001
10:38:19 6    DistributedActorsPackageTests.xctest 0x000055c85feb2958 <unavailable> + 3987800
10:38:19 7    DistributedActorsPackageTests.xctest 0x000055c85feb3b51 <unavailable> + 3992401
10:38:19 8    DistributedActorsPackageTests.xctest 0x000055c85fe876ea <unavailable> + 3811050
10:38:19 9    DistributedActorsPackageTests.xctest 0x000055c85fe85ee1 <unavailable> + 3804897
10:38:19 10   DistributedActorsPackageTests.xctest 0x000055c85fe8adff <unavailable> + 3825151
10:38:19 11   DistributedActorsPackageTests.xctest 0x000055c85fea4c9b <unavailable> + 3931291
10:38:19 12   DistributedActorsPackageTests.xctest 0x000055c85fe1133e <unavailable> + 3326782
10:38:19 13   DistributedActorsPackageTests.xctest 0x000055c86009725f <unavailable> + 5972575
10:38:19 14   DistributedActorsPackageTests.xctest 0x000055c860095dae <unavailable> + 5967278
10:38:19 15   DistributedActorsPackageTests.xctest 0x000055c8600959e6 <unavailable> + 5966310
10:38:19 16   DistributedActorsPackageTests.xctest 0x000055c85fdd8f66 <unavailable> + 3096422
10:38:19 17   DistributedActorsPackageTests.xctest 0x000055c85ffa4538 <unavailable> + 4977976
10:38:19 18   DistributedActorsPackageTests.xctest 0x000055c85ffa4936 <unavailable> + 4978998
10:38:19 19   DistributedActorsPackageTests.xctest 0x000055c85ffa2bd1 <unavailable> + 4971473
10:38:19 20   DistributedActorsPackageTests.xctest 0x000055c85ffa2831 <unavailable> + 4970545
10:38:19 21   DistributedActorsPackageTests.xctest 0x000055c85ffa2eb9 <unavailable> + 4972217
10:38:19 22   DistributedActorsPackageTests.xctest 0x000055c85fbe4554 <unavailable> + 1045844
10:38:19 23   DistributedActorsPackageTests.xctest 0x000055c85ffa968e <unavailable> + 4998798
10:38:19 24   DistributedActorsPackageTests.xctest 0x000055c85ffb2ef9 <unavailable> + 5037817
10:38:19 25   DistributedActorsPackageTests.xctest 0x000055c85fe0acac <unavailable> + 3300524
10:38:19 26   DistributedActorsPackageTests.xctest 0x000055c85ff96f71 <unavailable> + 4923249
10:38:19 27   DistributedActorsPackageTests.xctest 0x000055c85fe0b0dc <unavailable> + 3301596
10:38:19 28   DistributedActorsPackageTests.xctest 0x000055c85ff970c1 <unavailable> + 4923585
10:38:19 29   DistributedActorsPackageTests.xctest 0x000055c85ff969c1 <unavailable> + 4921793
10:38:19 30   DistributedActorsPackageTests.xctest 0x000055c85ff96b31 <unavailable> + 4922161
10:38:19 31   DistributedActorsPackageTests.xctest 0x000055c8600ab7b7 <unavailable> + 6055863
10:38:19 32   DistributedActorsPackageTests.xctest 0x000055c8600ab849 <unavailable> + 6056009
10:38:19 33   DistributedActorsPackageTests.xctest 0x000055c8600ac6d5 <unavailable> + 6059733
10:38:19 34   DistributedActorsPackageTests.xctest 0x000055c8600ac729 <unavailable> + 6059817
10:38:19 35   libpthread.so.0                    0x00007fe6b203e6db <unavailable> + 30427
10:38:19 36   libc.so.6                          0x00007fe6b04e8850 clone + 63
10:38:22 /code/Tests/DistributedActorsTests/Cluster/AssociationClusteredTests.swift:137: error: ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing : failed - 
10:38:22         _ = try p7337.expectMessage()
10:38:22                      ^~~~~~~~~~~~~~
10:38:22 error: Did not receive message of type [HandshakeResult] within [3s], error: noMessagesInQueue
10:38:22 <EXPR>:0: error: ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing : threw error "error(message: "\n        _ = try p7337.expectMessage()\n                     \u{1B}[0;31m^~~~~~~~~~~~~~\nerror: Did not receive message of type [HandshakeResult] within [3s], error: noMessagesInQueue\u{1B}[0;0m")"
10:38:22 ------------------------------------- ActorSystem(first, sact://first@localhost:9001) ------------------------------------------------
10:38:22 Captured log [first]: [trace] Started timer [TimerKey(receptionist/sync)] with generation [0]
10:38:22 Captured log [first]: [debug] Spawning [DistributedActors.Behavior<DistributedActors.DowningStrategyMessage>.setup((Function))], on path: [/system/cluster/downingStrategy]
10:38:22 Captured log [first]: [info] Binding to: [sact://first@localhost:9001]
10:38:22 Captured log [first]: [trace] Started timer [TimerKey(swim/periodic-ping)] with generation [0]
10:38:22 Captured log [first]: [trace] Successfully subscribed [ActorRef<ClusterEvent>(/system/cluster/downingStrategy/$sub-DistributedActors.ClusterEvent-y)]
10:38:22 Captured log [first]: [info] Bound to [IPv4]127.0.0.1/127.0.0.1:9001
10:38:22 Captured log [first]: [info] Extending handshake offer to sact://second@localhost:9002)
10:38:22 Captured log [first]: [warning] Failed await for outbound channel to sact://second@localhost:9002; Error was: ExecutionError(underlying: NIO.NIOConnectionError(host: "localhost", port: 9002, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv6]localhost/::1:9002, error: connect(descriptor:addr:size:) failed: Network is unreachable (errno: 101)), NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:9002, error: connection reset (error set): Connection refused (errno: 111))]))
10:38:22 Captured log [first]: [info] Schedule handshake retry to: [sact://second@localhost:9002] delay: [TimeAmount(100ms, nanoseconds: 100000000)]
10:38:22 Captured log [first]: [trace] Started timer [TimerKey(handshake-timer-sact://second@localhost:9002)] with generation [0]
10:38:22 Captured log [first]: [warning] Concurrently initiated handshakes from nodes [sact://first@localhost:9001](local) and [sact://second@localhost:9002](remote) detected! Resolving race by address ordering; This node WON (will negotiate and reply) tie-break. 
10:38:22 Captured log [first]: [warning] Supervision: Actor has FAULTED while processingUserMessages, handling with DistributedActors.StoppingSupervisor<DistributedActors.ClusterShell.Message>; Failure details: fault(Actor faulted while processing message '[inbound(DistributedActors.ClusterShell.InboundMessage.handshakeOffer(DistributedActors.Wire.HandshakeOffer(version: Version(0.0.1, reserved:0), from: sact://second:3987008798@localhost:9002, to: sact://first@localhost:9001), channel: SocketChannel { selectable = BaseSocket { fd=377 } , localAddress = Optional([IPv4]127.0.0.1/127.0.0.1:9001), remoteAddress = Optional([IPv4]127.0.0.1/127.0.0.1:34464) }, replyTo: NIO.EventLoopPromise<DistributedActors.Wire.HandshakeResponse>(futureResult: NIO.EventLoopFuture<DistributedActors.Wire.HandshakeResponse>)))]:Message':
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0xfff43) [0x55c85fbe4f43]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x100421) [0x55c85fbe5421]
10:38:22 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fe6b2049890]
10:38:22 /root/.swift/usr/lib/swift/linux/libswiftCore.so(+0x321aba) [0x7fe6b2577aba]
10:38:22 /root/.swift/usr/lib/swift/linux/libswiftCore.so(+0x1485e9) [0x7fe6b239e5e9]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3cd958) [0x55c85feb2958]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3ceb51) [0x55c85feb3b51]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3a26ea) [0x55c85fe876ea]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3a0ee1) [0x55c85fe85ee1]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3a5dff) [0x55c85fe8adff]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3bfc9b) [0x55c85fea4c9b]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x32c33e) [0x55c85fe1133e]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5b225f) [0x55c86009725f]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5b0dae) [0x55c860095dae]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5b09e6) [0x55c8600959e6]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x2f3f66) [0x55c85fdd8f66]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4bf538) [0x55c85ffa4538]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4bf936) [0x55c85ffa4936]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4bdbd1) [0x55c85ffa2bd1]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4bd831) [0x55c85ffa2831]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4bdeb9) [0x55c85ffa2eb9]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0xff554) [0x55c85fbe4554]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4c468e) [0x55c85ffa968e]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4cdef9) [0x55c85ffb2ef9]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x325cac) [0x55c85fe0acac]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4b1f71) [0x55c85ff96f71]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x3260dc) [0x55c85fe0b0dc]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4b20c1) [0x55c85ff970c1]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4b19c1) [0x55c85ff969c1]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x4b1b31) [0x55c85ff96b31]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5c67b7) [0x55c8600ab7b7]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5c6849) [0x55c8600ab849]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5c76d5) [0x55c8600ac6d5]
10:38:22 /code/.build/x86_64-unknown-linux/debug/DistributedActorsPackageTests.xctest(+0x5c7729) [0x55c8600ac729]
10:38:22 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fe6b203e6db]
10:38:22 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fe6b04e888f]
10:38:22 Captured log [first]: [error] Actor crashing, reason: [Actor faulted while processing message '[inbound(DistributedActors.ClusterShell.InboundMessage.handshakeOffer(DistributedActors.Wire.HandshakeOffer(version: Version(0.0.1, reserved:0), from: sact://second:3987008798@localhost:9002, to: sact://first@localhost:9001), channel: SocketChannel { selectable = BaseSocket { fd=377 } , localAddress = Optional([IPv4]127.0.0.1/127.0.0.1:9001), remoteAddress = Optional([IPv4]127.0.0.1/127.0.0.1:34464) }, replyTo: NIO.EventLoopPromise<DistributedActors.Wire.HandshakeResponse>(futureResult: NIO.EventLoopFuture<DistributedActors.Wire.HandshakeResponse>)))]:Message', with backtrace]:MessageProcessingFailure. Terminating actor, process and thread remain alive.
10:38:22 Captured log [first]: [trace] Cancel timer [TimerKey(handshake-timer-sact://second@localhost:9002)] with generation [0]
10:38:22 Captured log [first]: [trace] Received periodic trigger to ping random member
10:38:22 Captured log [first]: [trace] Received periodic trigger to ping random member
10:38:22 ========================================================================================================================
10:38:22 ------------------------------------- ActorSystem(second, sact://second@localhost:9002) ------------------------------------------------
10:38:22 Captured log [second]: [trace] Started timer [TimerKey(receptionist/sync)] with generation [0]
10:38:22 Captured log [second]: [debug] Spawning [DistributedActors.Behavior<DistributedActors.DowningStrategyMessage>.setup((Function))], on path: [/system/cluster/downingStrategy]
10:38:22 Captured log [second]: [info] Binding to: [sact://second@localhost:9002]
10:38:22 Captured log [second]: [trace] Started timer [TimerKey(swim/periodic-ping)] with generation [0]
10:38:22 Captured log [second]: [trace] Successfully subscribed [ActorRef<ClusterEvent>(/system/cluster/downingStrategy/$sub-DistributedActors.ClusterEvent-y)]
10:38:22 Captured log [second]: [info] Bound to [IPv4]127.0.0.1/127.0.0.1:9002
10:38:22 Captured log [second]: [info] Extending handshake offer to sact://first@localhost:9001)
10:38:22 Captured log [second]: [trace] Received periodic trigger to ping random member
10:38:22 Captured log [second]: [trace] Received periodic trigger to ping random member
10:38:22 ========================================================================================================================
10:38:22 Test Case 'ClusterAssociationTests.test_association_shouldEstablishSingleAssociationForConcurrentlyInitiatedHandshakes_incoming_outgoing' failed (3.23 seconds)

[ktoso]

Another instance in #47

My plan is to spend time on FAILED ones once the ProcessIsolated is in, context switching away from it is a bit painful.

// FYI @weissi @drexin

[ktoso]

Again in integration test breaking

22:38:10 Assertion failed: Channel should always be present after the initial initialization.: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 181
22:38:10 Current stack trace:
22:38:10 0    libswiftCore.so                    0x00007fbb5b57d7a0 _swift_stdlib_reportFatalErrorInFile + 115
22:38:10 1    libswiftCore.so                    0x00007fbb5b4b89cc <unavailable> + 3463628
22:38:10 2    libswiftCore.so                    0x00007fbb5b4b8abe <unavailable> + 3463870
22:38:10 3    libswiftCore.so                    0x00007fbb5b2b431a <unavailable> + 1348378
22:38:10 4    libswiftCore.so                    0x00007fbb5b48cab2 <unavailable> + 3283634
22:38:10 5    libswiftCore.so                    0x00007fbb5b2b35e9 <unavailable> + 1345001
22:38:10 6    it_ProcessIsolated_escalatingWorkers 0x0000563f460c83e8 <unavailable> + 3843048
22:38:10 7    it_ProcessIsolated_escalatingWorkers 0x0000563f460c95e1 <unavailable> + 3847649
22:38:10 8    it_ProcessIsolated_escalatingWorkers 0x0000563f4609e1ca <unavailable> + 3670474
22:38:10 9    it_ProcessIsolated_escalatingWorkers 0x0000563f4609c9b1 <unavailable> + 3664305
22:38:10 10   it_ProcessIsolated_escalatingWorkers 0x0000563f460a1bff <unavailable> + 3685375
22:38:10 11   it_ProcessIsolated_escalatingWorkers 0x0000563f460ba5ab <unavailable> + 3786155
22:38:10 12   it_ProcessIsolated_escalatingWorkers 0x0000563f4601730e <unavailable> + 3117838
22:38:10 13   it_ProcessIsolated_escalatingWorkers 0x0000563f462bc9ef <unavailable> + 5892591
22:38:10 14   it_ProcessIsolated_escalatingWorkers 0x0000563f462bb53e <unavailable> + 5887294
22:38:10 15   it_ProcessIsolated_escalatingWorkers 0x0000563f462bb176 <unavailable> + 5886326
22:38:10 16   it_ProcessIsolated_escalatingWorkers 0x0000563f45fdb9b6 <unavailable> + 2873782
22:38:10 17   it_ProcessIsolated_escalatingWorkers 0x0000563f461b6ad8 <unavailable> + 4819672
22:38:10 18   it_ProcessIsolated_escalatingWorkers 0x0000563f461b6ed6 <unavailable> + 4820694
22:38:10 19   it_ProcessIsolated_escalatingWorkers 0x0000563f461b5ec1 <unavailable> + 4816577
22:38:10 20   it_ProcessIsolated_escalatingWorkers 0x0000563f461b5b21 <unavailable> + 4815649
22:38:10 21   it_ProcessIsolated_escalatingWorkers 0x0000563f461b61a9 <unavailable> + 4817321
22:38:10 22   it_ProcessIsolated_escalatingWorkers 0x0000563f45de05e6 <unavailable> + 796134
22:38:10 23   it_ProcessIsolated_escalatingWorkers 0x0000563f461bc4b6 <unavailable> + 4842678
22:38:10 24   it_ProcessIsolated_escalatingWorkers 0x0000563f461c59b9 <unavailable> + 4880825
22:38:10 25   it_ProcessIsolated_escalatingWorkers 0x0000563f46010c7c <unavailable> + 3091580
22:38:10 26   it_ProcessIsolated_escalatingWorkers 0x0000563f461aa211 <unavailable> + 4768273
22:38:10 27   it_ProcessIsolated_escalatingWorkers 0x0000563f460110ac <unavailable> + 3092652
22:38:10 28   it_ProcessIsolated_escalatingWorkers 0x0000563f461aa361 <unavailable> + 4768609
22:38:10 29   it_ProcessIsolated_escalatingWorkers 0x0000563f461a9c61 <unavailable> + 4766817
22:38:10 30   it_ProcessIsolated_escalatingWorkers 0x0000563f461a9dd1 <unavailable> + 4767185
22:38:10 31   it_ProcessIsolated_escalatingWorkers 0x0000563f462d2ae7 <unavailable> + 5982951
22:38:10 32   it_ProcessIsolated_escalatingWorkers 0x0000563f462d2b79 <unavailable> + 5983097
22:38:10 33   it_ProcessIsolated_escalatingWorkers 0x0000563f462d3a05 <unavailable> + 5986821
22:38:10 34   it_ProcessIsolated_escalatingWorkers 0x0000563f462d3a59 <unavailable> + 5986905
22:38:10 35   libpthread.so.0                    0x00007fbb5af536db <unavailable> + 30427
22:38:10 36   libc.so.6                          0x00007fbb593fd850 clone + 63
22:38:10 ++ spin=4
22:38:10 ++ [[ 4 -eq 20 ]]

https://ci.swiftserver.group/job/swift-distributed-actors-swift50-p1rb/57/console

Going to focus on this one tomorrow unless someone comes first :)

[drexin]

Okay, after debugging this all day, I found a few issues:

1. In onOutboundConnectionError we don't abort the handshake when we schedule a retry. At that point the channel is not yet set, however, because that's what we are waiting for there. https://github.com/apple/swift-distributed-actors/blob/master/Sources/DistributedActors/Cluster/ClusterShell.swift#L472-L478

2. Also in onOutboundConnectionError, we set the state to initiated again, even if it should not be retried: https://github.com/apple/swift-distributed-actors/blob/master/Sources/DistributedActors/Cluster/ClusterShell.swift#L485

3. Even when fixing that, we crash when we eventually receive the handshake response from the other node: https://github.com/apple/swift-distributed-actors/blob/master/Sources/DistributedActors/Cluster/ClusterShellState.swift#L289-L291

I'm not exactly sure what the best way forward here is. @ktoso WDYT?