Zookeeper pods unable to see each other

[cdelpinogeo]

I am deploying the Zookeeper statefulset manifest and I see the following in the logs from the pods:

	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:166)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:595)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)
	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)
2017-06-16 02:29:03,164 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000
2017-06-16 02:30:03,165 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 2 at election address api-pers-zoo-1.capi.svc.cluster.local:3888
java.net.UnknownHostException: api-pers-zoo-1.capi.svc.cluster.local
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)
	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)
2017-06-16 02:30:03,169 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@173] - Failed to resolve address: api-pers-zoo-1.capi.svc.cluster.local
java.net.UnknownHostException: api-pers-zoo-1.capi.svc.cluster.local: Name does not resolve
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:166)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:595)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)
	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)

I've updated the command in the statefulset manifest to provide the complete name of each zookeeper pod:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: api-pers-zoo
  namespace: capi
spec:
  serviceName: "api-pers-zoo"
  replicas: 5
  template:
    metadata:
      labels:
        app: api-pers-zookeeper
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: api-pers-zookeeper
          image: solsson/zookeeper-statefulset
          env:
            - name: ZOO_SERVERS
              value: server.1=api-pers-zoo-0.capi.svc.cluster.local:2888:3888:participant server.2=api-pers-zoo-1.capi.svc.cluster.local:2888:3888:participant server.3=api-pers-zoo-2.capi.svc.cluster.local:2888:3888:participant server.4=api-pers-zoo-3.capi.svc.cluster.local:2888:3888:participant server.5=api-pers-zoo-4.capi.svc.cluster.local:2888:3888:participant
          ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: peer
            - containerPort: 3888
              name: leader-election
          volumeMounts:
            - name: datadir
              mountPath: /data
            # There's defaults in this folder, such as logging config
            #- name: conf
            #  mountPath: /conf
      volumes:
        #- name: conf
        #  emptyDir: {}
        - name: datadir
          emptyDir: {}

Services defined as follow:

apiVersion: v1
kind: Service
metadata:
  name: api-pers-zoo
  namespace: capi
spec:
  ports:
  - port: 2888
    name: peer
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: api-pers-zookeeper

and

# the headless service is for PetSet DNS, this one is for clients
apiVersion: v1
kind: Service
metadata:
  name: api-pers-zookeeper
  namespace: capi
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: api-pers-zookeeper

Any ideas where I am going wrong? Thanks!

[priscofarina]

Have you solved?

[cdelpinogeo]

Apologies for not updating the issue. Yes, this has been resolved. Thanks!

…

On Tue, Jul 25, 2017 at 4:53 AM, darkprisco ***@***.***> wrote: Have you solved? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AId2wPhmGiOA_Vd9pLyk7DZnkZi8qFnRks5sRa0CgaJpZM4N79PL> .

-- Christian Del Pino DevOps Engineer Boundless | www.boundlessgeo.com Email: cdelpino@boundlessgeo.com

[priscofarina]

It should be useful if you post how you have solved

[cdelpinogeo]

Sure thing. Here is the statefulset manifest:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: zoo
spec:
  serviceName: "zoo"
  replicas: 5
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: api-zookeeper
          image: solsson/zookeeper-statefulset:3.4.9
          env:
            - name: ZOO_SERVERS
              value: server.1=zoo-0.zoo:2888:3888:participant server.2=zoo-1.zoo:2888:3888:participant server.3=zoo-2.zoo:2888:3888:participant server.4=zoo-3.zoo:2888:3888:participant server.5=zoo-4.zoo:2888:3888:participant
          ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: peer
            - containerPort: 3888
              name: leader-election
          volumeMounts:
            - name: datadir
              mountPath: /data
            # There's defaults in this folder, such as logging config
            #- name: conf
            #  mountPath: /conf
      volumes:
        #- name: conf
        #  emptyDir: {}
        - name: datadir
          emptyDir: {}

[priscofarina]

thank you :-)

[solsson]

Since https://github.com/Yolean/kubernetes-kafka/releases/tag/v2.0.0 the zookeeper servers config is in the configmap, https://github.com/Yolean/kubernetes-kafka/blob/v2.0.0/zookeeper/10zookeeper-config.yml#L16, with modification of self's name in the init script, https://github.com/Yolean/kubernetes-kafka/blob/v2.0.0/zookeeper/10zookeeper-config.yml#L14. I think this is adequate, so we don't need to venture into Zookeeper 3.5+ dynamic config.

[sibtainabbas10]

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: zoo
spec:
serviceName: "zoo"
replicas: 5
template:
metadata:
labels:
app: zookeeper
spec:
terminationGracePeriodSeconds: 10
containers:
- name: api-zookeeper
image: solsson/zookeeper-statefulset:3.4.9
env:
- name: ZOO_SERVERS
value: server.1=zoo-0.zoo:2888:3888:participant server.2=zoo-1.zoo:2888:3888:participant server.3=zoo-2.zoo:2888:3888:participant server.4=zoo-3.zoo:2888:3888:participant server.5=zoo-4.zoo:2888:3888:participant
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: peer
- containerPort: 3888
name: leader-election
volumeMounts:
- name: datadir
mountPath: /data
# There's defaults in this folder, such as logging config
#- name: conf
# mountPath: /conf
volumes:
#- name: conf
# emptyDir: {}
- name: datadir
emptyDir: {}

@cdelpinogeo Hi, can you please tell me what is "zoo-0" and "zoo" in server.1=zoo-0.zoo:2888:3888:participant. Thanks

[solsson]

@sibtainabbas10 I think zoo is the "serviceName", see https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ and https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

[sibtainabbas10]

@solsson Thanks for the reply. I tried @cdelpinogeo's solution but it didn't work for me. Following is what I tried just like @cdelpinogeo suggested:

---
apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 2
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        image: solsson/zookeeper-statefulset:3.4.9
        env:
        - name: ZOO_SERVERS
          value: server.1=zk-0.zk-hs:2888:3888:participant server.2=zk-1.zk-hs:2888:3888:participant
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        volumeMounts:
        - name: zk-claim
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 0
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: zk-claim
    spec:
      storageClassName: standard
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[solsson]

Looks very different from the current manifest in this repo. I think you'll have to dig into logs and zookeeper specifics.

[cdelpinogeo]

I used an older manifest @sibtainabbas10, in conjunction with the relevant service and kafka manifests. For zookeeper, the example I posted above is what works for me. Let me know if you have any questions.

[sibtainabbas10]

@cdelpinogeo I am now using 1 replica for zookeeper and the issue didn't seem to reproduce. But strangely my kafka cannot connect to the zookeeper. I provided --override zookeeper.connect=zk-0.zk-hs.default.svc.cluster.local:2181 in my kafka manifest, even tried using --override zookeeper.connect=zk-cs:2181. Any help would be appreciated. Thanks

[cdelpinogeo]

Apologies for the late reply @sibtainabbas10. Are you still having issues with this?