Client ignores deadline option for readStream operation

[antonmos]

🐛 Current behavior

Due to code here and here
the client does not honor thedefaultDeadline or deadline options.

Inability to set a timeout is a critical problem since it can lead to resource exhaustion during load spikes.

🔍 Steps to reproduce

See code linked above

Reproducible link

See code linked above

💭 Expected behavior

client honors deadline options.

Package version

all

KurrentDB Version

v24.10

Connection string

esdb://localhost:2113?tls=false

☁️ Deployment Environment

Single-node (Docker)

Other Deployment Details

No response

Operating system

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[linear]

DEV-762 Client ignores deadline option for readStream operation

[w1am]

Hey @antonmos

For reads and subscriptions, we do not use the deadline value at all, as it is set to infinity by default even though it is available as an option in the ReadStreamOptions class. The server is capable of sustaining long intervals without sending data to the client.

[linear]

from @alexeyzimarev:

Setting the deadline for reads is relevant because the server cancels expired reads in the queue to avoid useless reads when the client already dropped off by timeout. Subscriptions, however, should still have set the deadline to infinity.

[antonmos]

Hey @antonmos

For reads and subscriptions, we do not use the deadline value at all, as it is set to infinity by default even though it is available as an option in the ReadStreamOptions class. The server is capable of sustaining long intervals without sending data to the client.

1. the purpose of timeouts on read operation is to protect the resources of the client, no the server. For example, in case of network partition between client and server the client should fail fast instead of tying up its resources for a long time.
2. The ReadStreamOption API has the deadline option, which is is misleading and error prone, because the users dont know that they have to add another layer of timeouts to achieve item 1.

Based on @alexeyzimarev comment, could you confirm if this will be fixed?

[alexeyzimarev]

We'll fix it. Indeed, the options class has this field and it should be supported.

I am not sure about the first point though as the deadline is passed over to the server so it knows when the client stops waiting. In case of network partition between the client and the server, the connection will be closed and won't open again, so all the calls will be terminated. I am not sure if the gRPC call deadline somehow affects the client trying to establish connection to the server when the server simply cannot be reached. I believe the call will be terminated immediately or will be cancelled by the client itself.

According to gRPC docs:

If a server has gone past the deadline when processing a request, the client will give up and fail the RPC with the DEADLINE_EXCEEDED status.

which implies the client can reach the server but the server is too busy so it can't process the request in time

[antonmos]

in my testing, when the server goes down, and the client makes a new readStream call, the client takes 30+s to timeout. This is how i discovered the issue - these tests were taking way too long.

[linear]

from @alexeyzimarev:

Ok, that is weird. It might be a Java-specific gRPC issue or there's something else in our client. For example, the .NET client simply fails immediately if there's no connection.

[antonmos]

I was able to set up tests for this using test containers

 private void unpauseESDB() {
           eventStoreDbTestContainer.getDockerClient().unpauseContainerCmd(eventStoreDbTestContainer.getContainerId()).exec();
    }

    private void pauseESDB() {
        eventStoreDbTestContainer.getDockerClient().pauseContainerCmd(eventStoreDbTestContainer.getContainerId()).exec();
    }

[w1am]

Hey @antonmos We've fixed the issue with the read operation, and the update is available in the latest release on Maven. For any other issues, please open a new one