FAILED: test_singletonByClusterLeadership_stashMessagesIfNoLeader; "handshake which was not in progress!"

[ktoso]

Fatal error: Accept incoming for handshake which was not in progress!: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 329

Test Case 'ActorSingletonPluginClusteredTests.test_singletonByClusterLeadership_stashMessagesIfNoLeader' started at 2020-03-03 02:09:09.793
Fatal error: Accept incoming for handshake which was not in progress!: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 329
Current stack trace:
0    libswiftCore.so                    0x00007ff85fa663a0 swift_reportError + 50
1    libswiftCore.so                    0x00007ff85fad7090 _swift_stdlib_reportFatalErrorInFile + 115
2    libswiftCore.so                    0x00007ff85f9fb4fe <unavailable> + 3515646
3    libswiftCore.so                    0x00007ff85f9fb677 <unavailable> + 3516023
4    libswiftCore.so                    0x00007ff85f7f0ddd <unavailable> + 1375709
5    libswiftCore.so                    0x00007ff85f9d2458 <unavailable> + 3347544
6    libswiftCore.so                    0x00007ff85f7f0179 <unavailable> + 1372537
7    swift-distributed-actorsPackageTests.xctest 0x00005652f11e7d83 <unavailable> + 6700419
8    swift-distributed-actorsPackageTests.xctest 0x00005652f11a6626 <unavailable> + 6432294
9    swift-distributed-actorsPackageTests.xctest 0x00005652f11a2998 <unavailable> + 6416792
10   swift-distributed-actorsPackageTests.xctest 0x00005652f11ad324 <unavailable> + 6460196
11   swift-distributed-actorsPackageTests.xctest 0x00005652f11cf154 <unavailable> + 6598996
12   swift-distributed-actorsPackageTests.xctest 0x00005652f1016348 <unavailable> + 4793160
13   swift-distributed-actorsPackageTests.xctest 0x00005652f14776e6 <unavailable> + 9385702
14   swift-distributed-actorsPackageTests.xctest 0x00005652f1475d71 <unavailable> + 9379185
15   swift-distributed-actorsPackageTests.xctest 0x00005652f14759c1 <unavailable> + 9378241
16   swift-distributed-actorsPackageTests.xctest 0x00005652f0fc9bab <unavailable> + 4479915
17   swift-distributed-actorsPackageTests.xctest 0x00005652f1353fd9 <unavailable> + 8191961
18   swift-distributed-actorsPackageTests.xctest 0x00005652f134f60a <unavailable> + 8173066
19   swift-distributed-actorsPackageTests.xctest 0x00005652f10106ec <unavailable> + 4769516
20   swift-distributed-actorsPackageTests.xctest 0x00005652f1331501 <unavailable> + 8049921
21   swift-distributed-actorsPackageTests.xctest 0x00005652f1010aec <unavailable> + 4770540
22   swift-distributed-actorsPackageTests.xctest 0x00005652f1330f81 <unavailable> + 8048513
23   swift-distributed-actorsPackageTests.xctest 0x00005652f148cac7 <unavailable> + 9472711
24   swift-distributed-actorsPackageTests.xctest 0x00005652f148d778 <unavailable> + 9475960
25   swift-distributed-actorsPackageTests.xctest 0x00005652f148d829 <unavailable> + 9476137
26   libpthread.so.0                    0x00007ff85f48b6ba <unavailable> + 30394
27   libc.so.6                          0x00007ff85d3b43b0 clone + 109
0x5652f0da98a9, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /code/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:53
0x5652f0da98c8, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /code/<compiler-generated>:0
0x7ff85f49538f
0x7ff85f9d2465
0x7ff85f7f0178
0x5652f11e7d82, DistributedActors.ClusterShellState.incomingHandshakeAccept(DistributedActors.Wire.HandshakeAccept) -> Swift.Optional<DistributedActors.HandshakeStateMachine.CompletedState> at /code/Sources/DistributedActors/Cluster/ClusterShellState.swift:329
0x5652f11a6625, DistributedActors.ClusterShell.(onHandshakeAccepted in _574729D23FE177584A8E9754C8075C1C)(_: DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, _: DistributedActors.ClusterShellState, _: DistributedActors.Wire.HandshakeAccept, channel: NIO.Channel) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:783
0x5652f11a2997, receiveInbound #1 (_: DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, message: DistributedActors.ClusterShell.InboundMessage) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:458
0x5652f11ad323, closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, DistributedActors.ClusterShell.Message) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:542
0x5652f11cf153, partial apply forwarder for closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, DistributedActors.ClusterShell.Message) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/<compiler-generated>:0
0x5652f1016347, DistributedActors.Behavior.interpretMessage(context: DistributedActors.ActorContext<A>, message: A, file: Swift.StaticString, line: Swift.UInt) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Behaviors.swift:501
0x5652f14776e5, DistributedActors.Supervisor.interpretSupervised0(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, processingAction: DistributedActors.ProcessingAction<A>, nFoldFailureDepth: Swift.Int) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:468
0x5652f1475d70, DistributedActors.Supervisor.interpretSupervised0(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, processingAction: DistributedActors.ProcessingAction<A>) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:457
0x5652f14759c0, DistributedActors.Supervisor.interpretSupervised(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, message: A) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:421
0x5652f0fc9baa, DistributedActors.ActorShell.interpretMessage(message: A) throws -> DistributedActors.ActorRunResult at /code/Sources/DistributedActors/ActorShell.swift:285
0x5652f1353fd8, DistributedActors.Mailbox.(mailboxRun in _F5A1B51A24FD9930199EBF91BB3C2562)(DistributedActors.ActorShell<A>) -> DistributedActors.Mailbox<A>.(MailboxRunResult in _F5A1B51A24FD9930199EBF91BB3C2562) at /code/Sources/DistributedActors/Mailbox.swift:437
0x5652f134f609, DistributedActors.Mailbox.run() -> () at /code/Sources/DistributedActors/Mailbox.swift:331
0x5652f10106eb, reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /code/<compiler-generated>:0
0x5652f1331500, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /code/<compiler-generated>:0
0x5652f1010aeb, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out ()) to @escaping @callee_guaranteed () -> () at /code/<compiler-generated>:0
0x5652f1330f80, closure #1 () -> () in DistributedActors.FixedThreadPool.init(Swift.Int) throws -> DistributedActors.FixedThreadPool at /code/Sources/DistributedActors/FixedThreadPool.swift:75
0x5652f148cac6, closure #1 () -> () in DistributedActors.Thread.init(() -> ()) throws -> DistributedActors.Thread at /code/Sources/DistributedActors/Thread.swift:53
0x5652f148d777, closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static DistributedActors.Thread.runnerCallback.getter : (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> at /code/Sources/DistributedActors/Thread.swift:132
0x5652f148d828, @objc closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static DistributedActors.Thread.runnerCallback.getter : (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> at /code/<compiler-generated>:0
0x7ff85f48b6b9
0x7ff85d3b441c
0xffffffffffffffff
Exited with signal code 4
Build step 'Execute shell' marked build as failure
$ ssh-agent -k

[ktoso]

16:49:34 Test Case 'DowningClusteredTests.test_stopLeader_by_downSelf_shouldPropagateToOtherNodes' started at 2020-03-02 16:49:34.728
16:49:37 Fatal error: Accept incoming for handshake which was not in progress!: file /code/Sources/DistributedActors/Cluster/ClusterShellState.swift, line 329
16:49:37 Current stack trace:
16:49:37 0    libswiftCore.so                    0x00007f4270e603a0 swift_reportError + 50
16:49:37 1    libswiftCore.so                    0x00007f4270ed1090 _swift_stdlib_reportFatalErrorInFile + 115
16:49:37 2    libswiftCore.so                    0x00007f4270df54fe <unavailable> + 3515646
16:49:37 3    libswiftCore.so                    0x00007f4270df5677 <unavailable> + 3516023
16:49:37 4    libswiftCore.so                    0x00007f4270beaddd <unavailable> + 1375709
16:49:37 5    libswiftCore.so                    0x00007f4270dcc458 <unavailable> + 3347544
16:49:37 6    libswiftCore.so                    0x00007f4270bea179 <unavailable> + 1372537
16:49:37 7    swift-distributed-actorsPackageTests.xctest 0x0000558c304f7c13 <unavailable> + 6638611
16:49:37 8    swift-distributed-actorsPackageTests.xctest 0x0000558c304b6e16 <unavailable> + 6372886
16:49:37 9    swift-distributed-actorsPackageTests.xctest 0x0000558c304b3188 <unavailable> + 6357384
16:49:37 10   swift-distributed-actorsPackageTests.xctest 0x0000558c304bdb14 <unavailable> + 6400788
16:49:37 11   swift-distributed-actorsPackageTests.xctest 0x0000558c304df494 <unavailable> + 6538388
16:49:37 12   swift-distributed-actorsPackageTests.xctest 0x0000558c30336e88 <unavailable> + 4800136
16:49:37 13   swift-distributed-actorsPackageTests.xctest 0x0000558c307a60b6 <unavailable> + 9449654
16:49:37 14   swift-distributed-actorsPackageTests.xctest 0x0000558c307a4741 <unavailable> + 9443137
16:49:37 15   swift-distributed-actorsPackageTests.xctest 0x0000558c307a4391 <unavailable> + 9442193
16:49:37 16   swift-distributed-actorsPackageTests.xctest 0x0000558c302eab0b <unavailable> + 4487947
16:49:37 17   swift-distributed-actorsPackageTests.xctest 0x0000558c30680199 <unavailable> + 8245657
16:49:37 18   swift-distributed-actorsPackageTests.xctest 0x0000558c3067b7ca <unavailable> + 8226762
16:49:37 19   swift-distributed-actorsPackageTests.xctest 0x0000558c3033122c <unavailable> + 4776492
16:49:37 20   swift-distributed-actorsPackageTests.xctest 0x0000558c3065d6c1 <unavailable> + 8103617
16:49:37 21   swift-distributed-actorsPackageTests.xctest 0x0000558c3033162c <unavailable> + 4777516
16:49:37 22   swift-distributed-actorsPackageTests.xctest 0x0000558c3065d141 <unavailable> + 8102209
16:49:37 23   swift-distributed-actorsPackageTests.xctest 0x0000558c307bb497 <unavailable> + 9536663
16:49:37 24   swift-distributed-actorsPackageTests.xctest 0x0000558c307bc148 <unavailable> + 9539912
16:49:37 25   swift-distributed-actorsPackageTests.xctest 0x0000558c307bc1f9 <unavailable> + 9540089
16:49:37 26   libpthread.so.0                    0x00007f42708856ba <unavailable> + 30394
16:49:37 27   libc.so.6                          0x00007f426e7ad3b0 clone + 109
16:49:38 0x558c300c93e9, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /code/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:53
16:49:38 0x558c300c9408, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install() -> () at /code/<compiler-generated>:0
16:49:38 0x7f427088f38f
16:49:38 0x7f4270dcc465
16:49:38 0x7f4270bea178
16:49:38 0x558c304f7c12, DistributedActors.ClusterShellState.incomingHandshakeAccept(DistributedActors.Wire.HandshakeAccept) -> Swift.Optional<DistributedActors.HandshakeStateMachine.CompletedState> at /code/Sources/DistributedActors/Cluster/ClusterShellState.swift:329
16:49:38 0x558c304b6e15, DistributedActors.ClusterShell.(onHandshakeAccepted in _574729D23FE177584A8E9754C8075C1C)(_: DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, _: DistributedActors.ClusterShellState, _: DistributedActors.Wire.HandshakeAccept, channel: NIO.Channel) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:783
16:49:38 0x558c304b3187, receiveInbound #1 (_: DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, message: DistributedActors.ClusterShell.InboundMessage) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:458
16:49:38 0x558c304bdb13, closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, DistributedActors.ClusterShell.Message) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/Sources/DistributedActors/Cluster/ClusterShell.swift:542
16:49:38 0x558c304df493, partial apply forwarder for closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>, DistributedActors.ClusterShell.Message) throws -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in closure #1 (DistributedActors.ActorContext<DistributedActors.ClusterShell.Message>) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> in DistributedActors.ClusterShell.(ready in _574729D23FE177584A8E9754C8075C1C)(state: DistributedActors.ClusterShellState) -> DistributedActors.Behavior<DistributedActors.ClusterShell.Message> at /code/<compiler-generated>:0
16:49:38 0x558c30336e87, DistributedActors.Behavior.interpretMessage(context: DistributedActors.ActorContext<A>, message: A, file: Swift.StaticString, line: Swift.UInt) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Behaviors.swift:501
16:49:38 0x558c307a60b5, DistributedActors.Supervisor.interpretSupervised0(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, processingAction: DistributedActors.ProcessingAction<A>, nFoldFailureDepth: Swift.Int) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:468
16:49:38 0x558c307a4740, DistributedActors.Supervisor.interpretSupervised0(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, processingAction: DistributedActors.ProcessingAction<A>) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:457
16:49:38 0x558c307a4390, DistributedActors.Supervisor.interpretSupervised(target: DistributedActors.Behavior<A>, context: DistributedActors.ActorContext<A>, message: A) throws -> DistributedActors.Behavior<A> at /code/Sources/DistributedActors/Supervision.swift:421
16:49:38 0x558c302eab0a, DistributedActors.ActorShell.interpretMessage(message: A) throws -> DistributedActors.ActorRunResult at /code/Sources/DistributedActors/ActorShell.swift:286
16:49:38 0x558c30680198, DistributedActors.Mailbox.(mailboxRun in _F5A1B51A24FD9930199EBF91BB3C2562)(DistributedActors.ActorShell<A>) -> DistributedActors.Mailbox<A>.(MailboxRunResult in _F5A1B51A24FD9930199EBF91BB3C2562) at /code/Sources/DistributedActors/Mailbox.swift:437
16:49:38 0x558c3067b7c9, DistributedActors.Mailbox.run() -> () at /code/Sources/DistributedActors/Mailbox.swift:331
16:49:38 0x558c3033122b, reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /code/<compiler-generated>:0
16:49:38 0x558c3065d6c0, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at /code/<compiler-generated>:0
16:49:38 0x558c3033162b, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out ()) to @escaping @callee_guaranteed () -> () at /code/<compiler-generated>:0
16:49:38 0x558c3065d140, closure #1 () -> () in DistributedActors.FixedThreadPool.init(Swift.Int) throws -> DistributedActors.FixedThreadPool at /code/Sources/DistributedActors/FixedThreadPool.swift:75
16:49:38 0x558c307bb496, closure #1 () -> () in DistributedActors.Thread.init(() -> ()) throws -> DistributedActors.Thread at /code/Sources/DistributedActors/Thread.swift:53
16:49:38 0x558c307bc147, closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static DistributedActors.Thread.runnerCallback.getter : (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> at /code/Sources/DistributedActors/Thread.swift:132
16:49:38 0x558c307bc1f8, @objc closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static DistributedActors.Thread.runnerCallback.getter : (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> at /code/<compiler-generated>:0
16:49:38 0x7f42708856b9
16:49:38 0x7f426e7ad41c
16:49:38 0xffffffffffffffff
16:49:38 Exited with signal code 4

[ktoso]

This is pretty bad, should not be happening. Will add better assertions right now and debug more soon

[ktoso]

It might actually be just outright wrong to crash on this and we should just reject.

Actually, otherwise people could spoof a handshake and make us crash this way very easily