Better handle ensureAssociation/connect with dead node

[ktoso]

2020-02-14T11:00:23-0800 warning: [sact://SampleCluster@localhost:9111][ClusterShell.swift:740][thread:2331386][/system/cluster] Failed await for outbound channel to sact://SampleCluster@localhost:9333; Error was: NIOConnectionError(host: "localhost", port: 9333, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]localhost/127.0.0.1:9333, error: connection reset (error set): Connection refused (errno: 61))])

Today we don't backoff and keep trying forever, there should be a max limit and backoff there.

(Likely has some issue and definitely has TODOs in code)

[ktoso]

Example how this retries aggressively now:

// "tag": membership
Captured log [fourth][2020-04-29 0:2:42.9470] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:42.9480] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:42.9490] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(85ms 374μs, nanoseconds: 85374633)]
Captured log [fourth][2020-04-29 0:2:43.0390] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.0410] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.0410] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(115ms 971μs, nanoseconds: 115971704)]
Captured log [fourth][2020-04-29 0:2:43.1670] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.1680] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.1680] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(115ms 390μs, nanoseconds: 115390341)]
Captured log [fourth][2020-04-29 0:2:43.2950] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.2970] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.2970] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(117ms 599μs, nanoseconds: 117599442)]
Captured log [fourth][2020-04-29 0:2:43.4170] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.4180] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.4180] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(113ms 439μs, nanoseconds: 113439992)]
Captured log [fourth][2020-04-29 0:2:43.5400] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.5410] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.5410] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(76ms 954μs, nanoseconds: 76954668)]
Captured log [fourth][2020-04-29 0:2:43.6230] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.6240] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.6240] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(103ms 935μs, nanoseconds: 103935574)]
Captured log [fourth][2020-04-29 0:2:43.7380] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.7390] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.7390] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(116ms 742μs, nanoseconds: 116742061)]
Captured log [fourth][2020-04-29 0:2:43.8660] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.8680] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.8680] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(94ms 395μs, nanoseconds: 94395750)]
Captured log [fourth][2020-04-29 0:2:43.9670] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:43.9690] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:43.9690] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(87ms 465μs, nanoseconds: 87465173)]
Captured log [fourth][2020-04-29 0:2:44.0650] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.0660] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.0660] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(88ms 867μs, nanoseconds: 88867982)]
Captured log [fourth][2020-04-29 0:2:44.1620] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.1630] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.1630] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(89ms 235μs, nanoseconds: 89235100)]
Captured log [fourth][2020-04-29 0:2:44.2530] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.2550] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.2550] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(91ms 461μs, nanoseconds: 91461681)]
Captured log [fourth][2020-04-29 0:2:44.3510] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.3520] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.3520] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(98ms 637μs, nanoseconds: 98637606)]
Captured log [fourth][2020-04-29 0:2:44.4620] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.4630] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.4630] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(84ms 286μs, nanoseconds: 84286972)]
Captured log [fourth][2020-04-29 0:2:44.5560] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.5570] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.5570] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(109ms 541μs, nanoseconds: 109541079)]
Captured log [fourth][2020-04-29 0:2:44.6780] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.6800] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.6800] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(80ms 571μs, nanoseconds: 80571011)]
Captured log [fourth][2020-04-29 0:2:44.7660] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.7680] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.7680] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(82ms 79μs, nanoseconds: 82079055)]
Captured log [fourth][2020-04-29 0:2:44.8570] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.8580] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.8580] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(89ms 324μs, nanoseconds: 89324757)]
Captured log [fourth][2020-04-29 0:2:44.9560] [ClusterShell.swift:687][/system/cluster] [info] Extending handshake offer to sact://first@127.0.0.1:7111)
Captured log [fourth][2020-04-29 0:2:44.9580] [ClusterShell.swift:813][/system/cluster] [warning] Failed await for outbound channel to sact://first@127.0.0.1:7111; Error was: NIOConnectionError(host: "127.0.0.1", port: 7111, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]127.0.0.1/127.0.0.1:7111, error: connection reset (error set): Connection refused (errno: 61))])
Captured log [fourth][2020-04-29 0:2:44.9580] [ClusterShell.swift:824][/system/cluster] [debug] Schedule handshake retry to: [sact://first@127.0.0.1:7111] delay: [TimeAmount(84ms 224μs, nanoseconds: 84224592)]

[ktoso]

Solved as part of #604