Kafka comparing ip addresses with certs under unkown cercumstances

[vstoppe]

I am investigating for too long a strimzi-kafka installation and why the zookeeper compares the ip address of the target system´s certificate common / alt name. In this way the communication fails. This has been discussed in at least three issues:

• Broker cannot connect to Zookeeper #3111

• [Question] Zookeeper failed to verify host address after upgraded to 0.18.0 #3099

• Kafka PODs are not appearing, only zookeeper pods are running. #3155

• In one cluster the zookeeper communication fails:

  • Provider: RKE
  • Kubernetes v1.19.12
  • strimzi-kafka-version: 0.26.0
  • Here I don't have full access to the cluster.

• In another cluster it works fine:

  • Provider: K3(O)S:
  • Kubernetes: v.19.11+k3s1
  • strimzi-kafka-version: 0.27.0

I can get the zookeeper communication to work by setting the javaSystemProperties:

• zookeeper.ssl.hostnameVerification
• zookeeper.ssl.quorum.hostnameVerification

to false. But I would like to make it work without this workaround. Has anyone experienced this behaviour before, understood the problem and found a solution?

Here are some additional information:

2021-12-22 12:14:00,292 ERROR Failed to verify host address: 10.139.32.43 (org.apache.zookeeper.common.ZKTrustManager) [my-kafka-cluster-zookeeper-0.my-kafka-cluster-zookeeper-nodes.kafka-test.svc/10.42.4.240:3888]
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <10.139.32.43> doesn't match any of the subject alternative names: [my-kafka-cluster-zookeeper-client, my-kafka-cluster-zookeeper-client.kafka-test, *.my-kafka-cluster-zookeeper-nodes.kafka-test.svc.cluster.local, my-kafka-cluster-zookeeper-2.my-kafka-cluster-zookeeper-nodes.kafka-test.svc, *.my-kafka-cluster-zookeeper-client.kafka-test.svc.cluster.local, *.my-kafka-cluster-zookeeper-client.kafka-test.svc, *.my-kafka-cluster-zookeeper-nodes.kafka-test.svc, my-kafka-cluster-zookeeper-2.my-kafka-cluster-zookeeper-nodes.kafka-test.svc.cluster.local, my-kafka-cluster-zookeeper-client.kafka-test.svc, my-kafka-cluster-zookeeper-client.kafka-test.svc.cluster.local]

This log is interesting, because the ip make the following reverse lookup:

43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.rancher-monitoring-prometheus-node-exporter.cattle-monitoring-system.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.rancher-monitoring-ingress-nginx.ingress-nginx.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.rancher-monitoring-kubelet.kube-system.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.pushprox-kube-controller-manager-client.cattle-monitoring-system.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.pushprox-kube-proxy-client.cattle-monitoring-system.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.pushprox-kube-scheduler-client.cattle-monitoring-system.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.kubernetes.default.svc.cluster.local.
43.32.139.10.in-addr.arpa. 5    IN      PTR     10-139-32-43.pushprox-kube-etcd-client.cattle-monitoring-system.svc.cluster.local.

It must be a general ingress ip. I guess this is communication from the kafka-operator. Operator and Kafka run in different namespaces.

This is the configuration I deployed:

helm upgrade --install strimzi-eval strimzi/strimzi-kafka-operator -n kafka-operator -f values-operator.yaml

values-operator.yaml

# Default values for strimzi-kafka-operator.

# If you set `watchNamespaces` to the same value as ``.Release.Namespace` (e.g. `helm ... --namespace $NAMESPACE`),
# the chart will fail because duplicate RoleBindings will be attempted to be created in the same namespace
watchNamespaces: 
  - kafka-test
watchAnyNamespace: false




image:
  name: operator
  tag: 0.27.0
logVolume: co-config-volume
logConfigMap: strimzi-cluster-operator
logLevel: ${env:STRIMZI_LOG_LEVEL:-INFO}
fullReconciliationIntervalMs: 120000
operationTimeoutMs: 300000
kubernetesServiceDnsDomain: cluster.local
featureGates: ""

tolerations: []
affinity: {}
annotations: {}
labels: {}
nodeSelector: {}

podSecurityContext: {}
securityContext: {}

# Docker images that operator uses to provision various components of Strimzi.  To use your own registry prefix the
# repository name with your registry URL.
# Ex) repository: registry.xyzcorp.com/strimzi/zookeeper
zookeeper:
  image:
    name: kafka
    tagPrefix: 0.270
kafka:
  image:
    name: kafka
    tagPrefix: 0.27.0
kafkaConnect:
  image:
    name: kafka
    tagPrefix: 0.27.0
topicOperator:
  image:
    name: operator
    tag: 0.27.0
userOperator:
  image:
    name: operator
    tag: 0.27.0
kafkaInit:
  image:
    name: operator
    tag: 0.27.0
tlsSidecarEntityOperator:
  image:
    name: kafka
    tagPrefix: 0.27.0
kafkaMirrorMaker:
  image:
    name: kafka
    tagPrefix: 0.27.0
kafkaBridge:
  image:
    name: kafka-bridge
    tag: 0.20.3
kafkaExporter:
  image:
    name: kafka
    tagPrefix: 0.27.0
jmxTrans:
  image:
    name: jmxtrans
    tag: 0.27.0
kafkaMirrorMaker2:
  image:
    name: kafka
    tagPrefix: 0.27.0
cruiseControl:
  image:
    name: kafka
    tagPrefix: 0.27.0
tlsSidecarCruiseControl:
  image:
    name: kafka
    tagPrefix: 0.27  .0
kanikoExecutor:
  image:
    registry: gcr.io
    repository: kaniko-project
    name: kaniko-executor
    tag: v1.3.0
mavenBuilder:
  image:
    registry: ""
    repository: ""
    name: maven-builder
    tag: ""
resources:
  limits:
    memory: 384Mi
    cpu: 1000m
  requests:
    memory: 384Mi
    cpu: 200m
livenessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  initialDelaySeconds: 10
  periodSeconds: 30


createGlobalResources: true
# Override the exclude pattern for exclude some labels
labelsExclusionPattern: ""
# Controls whether Strimzi generates network policy resources (By default true)
generateNetworkPolicy: true
# Override the value for Connect build timeout
connectBuildTimeoutMs: 300000

In this way I deployed Kafka:

k apply -n kafka-test -f values-kafka.yaml

values-kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka-cluster
spec:
  clusterCa:
    generateCertificateAuthority: true
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      #- name: external
      #  port: 9094
      #  type: cluster-ip
      #  tls: true
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
        class: nfs-client
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.1.0"
      inter.broker.protocol.version: "2.1.0"
      ssl.truststore.password: "changeit"
      ssl.keystore.password: "changeit"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
      class: nfs-client
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

I think it is just how the ZooKeeper implements the hostname verification. It is probably much more secure ... but it also generates a lot more false negatives. But you can probably ask in some ZooKeeper forum to check on their exact motivation.

The reason why it somewhere works and somewhere not, is just in the networking configuration.

[vstoppe]

I wish someone had better hints. But I think I got something, but I need to talk with an k8s expert. Maybe this helps someone:

Even all zookeeper are in the name namespace, they are not in the same network. When I make a traceroute from zookeeper-0 to zookeeper-1 I see that the traffic takes some hops:

[kafka@my-kafka-cluster-zookeeper-0 kafka]$ ifconfig eth0 | grep -B 1 inet
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.4.45  netmask 255.255.255.255  broadcast 10.42.4.45

[kafka@my-kafka-cluster-zookeeper-0 kafka]$ tracepath my-kafka-cluster-zookeeper-1.my-kafka-cluster-zookeeper-nodes.kafka-test.svc.cluster.local
 1?: [LOCALHOST]                                         pmtu 1450
 1:  10-139-32-43.rancher-monitoring-prometheus-node-exporter.cattle-monitoring-system.svc.cluster.local   0.041ms
 1:  10-139-32-43.rancher-monitoring-prometheus-node-exporter.cattle-monitoring-system.svc.cluster.local   0.025ms
 2:  10.42.0.0                                             0.230ms
 3:  my-kafka-cluster-zookeeper-1.my-kafka-cluster-zookeeper-nodes.kafka-test.svc.cluster.local   0.214ms reached
     Resume: pmtu 1450 hops 3 back 3

It seems that k8s

• makes at 10.139.32.43 a NAT,
• the zookeeper tries to reverse lookup of the source ip address of the request,
• compares the wrong result with the cert,
• and brakes at the natted ip.

This is in the failing work environment. Unfortunately the images in my working private test environment have no tracepath, so it would be quiet time consuming to test it.
I suspect the problem is in the k8s network/nat settings.