Prometheus RSocket Proxy Client does not connect to Prometheus RSocket Proxy Server when using Spring Boot Native Images

[klopfdreh]

Currently we are facing the issue that if you use Spring Boot Native Images, the Prometheus RSocket Proxy Client does run into a timeout each time it connects to the Prometheus RSocket Proxy Server. For normal Java Application everything is working fine.

I also tried to increase the timeout with the newly introduced option #86

micrometer:
  prometheus:
    rsocket:
      timeout: 30s

Tested version: 2.0.0-M3

[klopfdreh]

I am going to upgrade to 2.0.0-RC1 and have a look if this fixes the issue.

Edit: The error still occurs with 2.0.0-RC1

[klopfdreh]

I applied the following logging configuration to debug:

logging:
  level:
    io.micrometer.prometheus.rsocket.PrometheusRSocketClient: trace

And saw the following output for a native application:

{"timestamp":"2025-03-03T08:56:03.687+0100","level":"DEBUG","thread":"main","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Connecting to RSocket Proxy...","context":"default"}
{"timestamp":"2025-03-03T08:56:43.692+0100","level":"WARN","thread":"main","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Creating the connection and receiving the key timed out!","context":"default"}

For the java application it was working fine all the time:

{"timestamp":"2025-03-03T08:58:24.826+0100","level":"DEBUG","thread":"main","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Connecting to RSocket Proxy...","context":"default"}
{"timestamp":"2025-03-03T08:58:25.469+0100","level":"INFO","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Connected to RSocket Proxy!","context":"default"}

Output with the recent changes in the case it is working:

{"timestamp":"2025-03-06T08:04:48.499+0100","level":"DEBUG","thread":"main","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Connecting to RSocket Proxy...","context":"default"}
{"timestamp":"2025-03-06T08:04:48.513+0100","level":"TRACE","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Acceptor for requestResponse and fireAndForget has been set up.","context":"default"}
{"timestamp":"2025-03-06T08:04:48.513+0100","level":"TRACE","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Successfully established connection to Prometheus proxy.","context":"default"}
{"timestamp":"2025-03-06T08:04:48.558+0100","level":"TRACE","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Received public key from Prometheus proxy in fireAndForget.","context":"default"}
{"timestamp":"2025-03-06T08:04:48.558+0100","level":"INFO","thread":"reactor-tcp-epoll-2","logger":"io.micrometer.prometheus.rsocket.PrometheusRSocketClient","message":"Connected to RSocket Proxy!","context":"default"}

I try to debug rsocket, now.

[klopfdreh]

I couldn't find the issue. spring-boot 3.4.1 is using rsocket-core 1.1.3 which supports native-image already: https://github.com/rsocket/rsocket-java/releases/tag/1.1.3

Note: The Java and Native application are configured exactly the same. (Port 7001 / Host <service>.<namespace>.svc.cluster.local / timeout 40s)

Maybe it would be good to upgrade the version to at lest 1.1.5 as there are updates to netty.

[klopfdreh]

I updated the native image setup and tested it again: https://github.com/klopfdreh/prometheus-rsocket-native-test - there was no issue at all, but for a local docker setup the service is missing in-between the connection from the client to the server - it seems to be some kind of a race condition.

[klopfdreh]

Updating to Spring Boot 3.4.3 and Prometheus RSocket Client 2.0.0-RC1 seems to improve this issue for native images. My assumption is that there has been some improvements

to Netty 4.1.116.Final compared to Netty 4.1.118.Final

• https://netty.io/news/2025/01/14/4-1-117-Final.html
• https://netty.io/news/2025/02/10/4-1-118-Final.html

or to RSocket

• https://github.com/rsocket/rsocket-java/releases/tag/1.1.4
• https://github.com/rsocket/rsocket-java/releases/tag/1.1.5

Also we decreased the Pod-Count to only 1 as during the TCP-Connection the key is exchanged which might not be available on the other Pod if the load balancing is round robin. I think it should be considered to write it into the documentation that you have to use either some kind of sticky interactions or only use one Pod.

Example for service:

kind: Service
apiVersion: v1
metadata:
 # ....
spec:
  # ....
  sessionAffinity: ClientIP

[klopfdreh]

I improved the client code a bit but I was not able to fix the issue within native images 100%. It is working most of the time, but sometimes it is like mentioned in the comment #94 (comment)

[klopfdreh]

When I set spring.threads.virtual.enabled=true and use Java 21 instead of Java 17 to compile the code the issue seems to be gone.