in 5.4.4, EventStoreDBClient becomes unusable after losing connection due to server is down

[antonmos]

This behavior started in version 5.4.4

Steps to reproduce

1. while server is up, start a subscription with client.subscribeToStream(streamName, subscriptionListener, subscriptionOptions).join()
2. shutdown server
3. in onCancelled handler, attempt to reconnecting using client.subscribeToStream(streamName, subscriptionListener, subscriptionOptions).join().

in 5.4.3

• CompletionException is thrown with cause com.eventstore.dbclient.ConnectionShutdownException: The connection is closed
• the onCancelled handler is not called

in 5.4.4,

• onCancelled handler is called with subscription parameter set to null
• no exception is thrown and the thread dies with the following lines written to the log (not via SLF4J logger)

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.71.0.jar:1.71.0]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:564) ~[grpc-stub-1.71.0.jar:1.71.0]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:565) ~[grpc-core-1.71.0.jar:1.71.0]
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.71.0.jar:1.71.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:733) ~[grpc-core-1.71.0.jar:1.71.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:714) ~[grpc-core-1.71.0.jar:1.71.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.71.0.jar:1.71.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.71.0.jar:1.71.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:2113
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:2113

Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
Caused by: java.net.ConnectException: Connection refused

	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[na:na]
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[grpc-netty-shaded-1.71.0.jar:1.71.0]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

[linear]

DEV-355 in 5.4.4, EventStoreDBClient becomes unusable after losing connection due to server is down

[w1am]

TODO: Increase test coverage for connection shutdown scenarios.