Zookeeper Readiness Probe Constantly Failing / Why no TCP Check

[elm-]

Hello,

I could not get the zookeeper part running consistently. I have random (about 1-2 per minute) a pod go into not ready for about 2-3 minutes, then back for a minute, then off again. I could not find the exact reason in the Kubernetes logs, just that the readiness probe failed. When I attach to the pod and run the nc command manually I always get a successful output, even when Kubernetes marked the pod as not ready. I tried increasing the timeout, but that was not the issue. Any other ideas how to further debug?

I'm running the latest version of the repo on 1.7.6 on GKE.

I changed it to a TCP readiness check and it works flawlessly. Only downside I get warnings in the logs because the client isn't sending proper commands, I can live with that. Is there any reason why not use a TCP probe as the default?

Cheers,
Elmar

[solsson]

The source of the zookeeper probe is here: 2c4b6cd

I logged #65 based on #63 and my own observations of this failing.

What does your TCP probe look like? The one I came up with for Kafka also pollutes the logs.

[elm-]

Yes, I used that version of the probe, with it I could not get a zookeeper instance ready for more than a minute. It was failing without any indication I could see. Running nc -w 127.0.0.1 2181 ; echo $? always outputs 0 when attaching, even when kubernetes said nope, so no idea why it doesn't work.

yes, it floods the logs with WARN in zookeeper (see below), but nothing in Kafka.

[2017-10-13 13:44:50,035] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:748)
[2017-10-13 13:45:00,034] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:748)

If it annoys you, I guess you could decrease the periodInterval in the probe or reconfigure the log level. We configure error and warnings rates in our monitoring and we can just ignore this individually so it would not trigger, so I don't see any issue there.

This is my TCP probe for Kafka:

        readinessProbe:
          tcpSocket:
            port: 9092
          timeoutSeconds: 10

Zookeeper same, just different ports. Timeout can also be lower I think if it's TCP based as it should always accept. From experience we set it a bit higher to not have it fail on random events like Garbage Collection, etc.

[elm-]

Addendum: Yes, I saw #65, did you try to run the command via exec when Kubernetes reported it as failing? Because for me that works, so I guess it's an issue with the probe, not the actual command. + Mine fails consistently every minute or so.

[elm-]

Didn't post this, so this is when I run the probe inside and I get very rarely back imok. However, the zookeeper process is OK the whole time.

imokroot@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
root@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -w 1 127.0.0.1 2181

[elm-]

Actually this works:

root@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka# echo ruok | nc -q 1 127.0.0.1 2181
imokroot@zoo-0:/opt/kafka#

-q makes it wait for a response after EOF

[elm-]

Created #74 as a suggestion to fix it.

[solsson]

It was actually github that closed this :) Please reopen if you think the TCP based probe is still preferrable.

[elm-]

For Kafka maybe, but for Zookeeper this makes more sense as it waits until the node is actually synchronized.