[Bug]: Kafka pods are not getting instantiated sporadiacally

[mibollma]

Bug Description

I am currently installing Kafka nightly on three stages on AKS and 95% of the time it is working as expected, about 5% of the time the install fails with the following situation:

• All Zookeeper pods are ready
• The StrimziPodSet kafka-stack-zookeeper exists and states that all zookeeper pods are ready
• The StrimziPodSet kafka-stack-kafka is missing
• All Kafka pods are missing
• The cluster operator logs the following warning hundreds of times until the installation reaches a timeout:

2023-12-15 07:54:12 WARN ClientCnxn:1300 - Session 0x0 for server kafka-stack-zookeeper-1.kafka-stack-zookeeper-nodes.kafka-stack-validation.svc/172.18.12.96:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
org.apache.zookeeper.ClientCnxn$EndOfStreamException: channel for sessionid 0x0 is lost
at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:286) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]

If I restart the cluster operator in this situation, everything starts working as expected and the Kafka pods are starting.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.38.0

Kubernetes version

Kubernetes 1.26.10

Installation method

Helm chart

Infrastructure

Azure AKS

Configuration files and logs

No response

Additional context

No response

[scholzj]

You have to share your configuration, the logs from all the pods etc. Without it, it looks like your ZOoKeeper does not work, but not much more can be said.

[mibollma]

To me it looks more like the strimzi operator is in a non-working state as restarting it resolves the issue and enables the operator to communicate with zookeeper.

[maschmid]

I think I may be seeing the same issue, occasionaly in our openshift knative CI , currently with strimzi 0.40.0

e.g. in this run https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-knative-eventing-kafka-broker-release-next-415-test-reconciler-aws-415-c/1780100520216629248

See the "kafka-" logs in

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-knative-eventing-kafka-broker-release-next-415-test-reconciler-aws-415-c/1780100520216629248/artifacts/test-reconciler-aws-415-c/openshift-gather-extra/artifacts/pods/

The strimzi operator install, Kafka and KafkaUser resources are created in https://github.com/openshift-knative/serverless-operator/blob/main/hack/lib/strimzi.bash

[scholzj]

Yeah, the logs suggests the ZooKeeper cluster did not form because of DNS issues (e.g. a slow DNS). Unfortunately, that is a known issue with ZooKeeper. One of the options might be to use KRaft (ZooKeeper-less Kafka) as Kafka seems to handle things such as slow DNS better and does not seem to suffer from the same issues. To do that, you would need to replace your current Kafka resource with something like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.0
    listeners:
      # PLAINTEXT
      - name: plain
        port: 9092
        type: internal
        tls: false
      # SSL
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      # protocol=SASL_SSL
      # sasl.mechanism=SCRAM-SHA-512
      - name: saslssl
        port: 9094
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
      # protocol=SASL_PLAINTEXT
      # sasl.mechanism=SCRAM-SHA-512
      - name: saslplain
        port: 9095
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      # TLS no auth
      - name: tlsnoauth
        port: 9096
        type: internal
        tls: true
    authorization:
      superUsers:
        - ANONYMOUS
      type: simple
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      inter.broker.protocol.version: "3.7"
      auto.create.topics.enable: "false"
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  namespace: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  resources:
    requests:
      memory: 2Gi
      cpu: "300m"
    limits:
      memory: 4Gi
      cpu: "4"

(I wrote it out of my head, so you should test it first 😉)

[jack1902]

I encountered the same issue above when running a scaled down kafka/zookeeper deployment within a kind cluster. I scaled coredns to 1 replica and things then came up happily.

I found the coredns fix from here kubernetes-sigs/kind#3713 (comment) which appears to be getting fixed elsewhere too