Fixes issue#398 that decreasing replicas will make zookeeper unrecoverable when zookeeper not running.

[stop-coding]

Change log description

Fixes the bug that decreasing replicas will make zookeeper unrecoverable when zookeeper not running.

Purpose of the change

Fixes #398

What the code does

Add protection for setting Stateful when zookeeper not running.
If zookeeper not running, we will prohibited to update replicas status until zookeeper resume.
When user decrease replicas value, it will remove node with reconfig firstly.
Keep do that remove node with reconfig on preStop before pod exit.

How to verify it

1. Create an cluster that size is 3 (kubectl create -f zk.yaml).
2. Wait all pod running, named: zk-0\zk-1\zk-2.
3. Delete zk-1\zk-2 pod, make cluster of zookeeper unable to provide services.
4. "kubectl edit zk" that change replicas to 1 immediately.
5. Wait some time, replicas will decrease to 1.
6. Now, checking that:
   Is zk-0 is all right?

[codecov]

Codecov Report

Merging #399 (15e58cf) into master (e6d84b8) will decrease coverage by 0.07%.
The diff coverage is 85.18%.

@@            Coverage Diff             @@
##           master     #399      +/-   ##
==========================================
- Coverage   84.11%   84.04%   -0.08%     
==========================================
  Files          12       12              
  Lines        1643     1667      +24     
==========================================
+ Hits         1382     1401      +19     
- Misses        177      185       +8     
+ Partials       84       81       -3     

Impacted Files	Coverage Δ
pkg/zk/zookeeper_client.go	82.50% <77.77%> (-1.38%)	⬇️
...er/zookeepercluster/zookeepercluster_controller.go	63.50% <88.88%> (+0.47%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6d84b8...15e58cf. Read the comment docs.

[anishakj]

If we wont compare with spec.Replicas status wont be shown correctly rt, in the case of scaling

[jkhalack]

I am afraid replacing instance.Spec.Replicas with instance.Status.Replicas won't work for initial ZK cluster deployment and scale up scenarios.
The very next line logging Cluster is Ready would not be true if we only compare to the status.

Remember, when the cluster is initially created, the pods are created one by one.
If we adopt instance.Status.Replicas == instance.Status.ReadyReplicas condition, then we will be moving to the next step after the first pod is created, not waiting for the rest of the replicas.

[stop-coding]

Yes, I get it.
This condition only take effect on scale up, so It won't affect shrinking.
Is it more appropriate that judging if the cluster is ready with the condition of
instance.Spec.Replicas == instance.Status.ReadyReplicas and instance.Spec.Replicas == instance.Status.Replicas ?

[anishakj]

@stop-coding have one more question. How you made the service of zookeeper cluster to stop? will it be possible to add a test for the same

[stop-coding]

@stop-coding have one more question. How you made the service of zookeeper cluster to stop? will it be possible to add a test for the same

@anishakj
Delete more than half pods of zookeeper after editing the replicas on zk immediately.
It will also happen that some node which pod is running have network failure when we edit the replicas.