Decreasing the replicas for kafka and zookeeper pods

[shreyasarani23]

1. Say I have deployed my kafka cluster with 3 replicas of kafka and zookeeper. And after that I have increased replicas to 4 for both kafka and zookeeper pods. And now I want to decrease the replicas to 3 for both kafka and zookeeper. While decreasing the replica does it remove the last created pod for kafka [my-kafka-cluster-kafka-3 in this case] or just remove some random kafka broker pod?
2. After decreasing the replicas, the persistent volume and pvc for that pod , still exists so is there a way to delete it or we have to manually do it?
3. After increasing the replicas, the new kafka broker doesn't contain any topic nor the data. So is there a way that this new kafka broker can get in touch with previously created kafka brokers and copy the topic details to the newly created kafka broker. If so how this can be achieved?

[scholzj]

First of all, you should always have an odd number of ZooKeeper replicas - for example 3 or 5. Zookeeper always needs a majority of nodes to have a quorum - so having 4 nodes makes only a little sense. (For Kafka, having 4 nodes might be fine)

Re 1) Scaling down always removes the last pod - the one with the highest index number.

Re 2) In general, Kubernetes leaves the deletion of the volumes to the user - as a protection against accidentally deleting the data. I think you should be able to set the deleteClaim flag in storage configuration to true to have volumes deleted automatically. But I would recommend that only for development. Not for production.

Re 3) Partition replicas are always assigned to specific brokers. If you want to move some of them to the new broker after scale-up, you can use the KafkaRebalance resource to rebalance the cluster. Check the docs about more info. Also, before scale down, you have to (manually) move the partition replicas from the broker which will be removed. Keep in mind that moving the partition-replicas takes some time. Kafka is not something what easily scales up and down every few minutes. It is more rare thing when you onboard new projects etc. for example.

[shreyasarani23]

Thanks @scholzj for the explanation.