Autonomous control plane

[majst01]

Description

@mwindower @robertvolkmann @Gerrit91 @vknabel @simcod first version, would love to get initial feedback if this is at least on the right track

TODO:

• Add a section which failures are possible and which services are affected.
• Add kind cluster upgrade description and test

[metal-robot]

Thanks for contributing a pull request to the metal-stack docs!

A rendered preview of your changes will be available at: https://docs.metal-stack.io/previews/PR235/

[robertvolkmann]

I struggle to understand why we want to deploy two separate single-node kind clusters instead of deploying a multi-node cluster. DRBD and pacemaker are new technologies for me, but a multi-node cluster is well-known for me.

My gut feeling leans more towards creating just a bootstrap cluster that will be deleted after everything is set up, but I didn't have time to think it through.

[majst01]

The two node cluster setup is just one possibility i proposed. I tried to give a minimal example which gives some sort of HA. This could of course be improved. The most important point here is that the etcd of of the kind cluster and the garden-api-server is backed up on a regular short interval to a external S3 storage. Loosing this kind cluster, or the machine it is executed leads to the inability to manage the control plane of the upper layer, but not the main partition.

[Gerrit91]

For now only typos. :D

[majst01]

For now only typos. :D

Thanks

[chbmuc]

Good proposal. Two considerations from my side:

• I'm a bit reluctant regarding the persistent storage for the needle nodes. I think we should prefer an external storage system over the suggested DRBD solution. I think it would be best to define a protocol (maybe something basic like iSCSI) that should be used and a set of volumes that should be provided by the storage system. This way we would not require a specific solution and let the datacenter provide the volumes on whatever system it has available. If there is no existing storage system, we still cloud go for a (synology) appliance.

• It is'n clear to me, what happens, when the kind node fails. Don't we need an extra failover mechanism? Or ist this something you wanted to solve with pacemaker?

[majst01]

Good proposal. Two considerations from my side:

• I'm a bit reluctant regarding the persistent storage for the needle nodes. I think we should prefer an external storage system over the suggested DRBD solution. I think it would be best to define a protocol (maybe something basic like iSCSI) that should be used and a set of volumes that should be provided by the storage system. This way we would not require a specific solution and let the datacenter provide the volumes on whatever system it has available. If there is no existing storage system, we still cloud go for a (synology) appliance.

The DRBD solution was only meant as a redundancy solution for the kind stateful sets like etcd, metal-db and other, but not for the needle nodes. For them i am completely with you that this should be provided from external. But OTOH it would also possible to use the local NVMe drives of the needle nodes for storage if we create the always shoots with HA-Control Plane.

• It is'n clear to me, what happens, when the kind node fails. Don't we need an extra failover mechanism? Or ist this something you wanted to solve with pacemaker?

If the kind node fails, the managed needle clusters are still working, but managing of them is not possible. Access to the needle seed is not possible, but needle shoot access is.

I need to add a description of possible failure scenarios to make this clearer

[vknabel]

Great summary, though it feels a little bit complicated for the fact that we are trying to eat our own dog food. Just a feeling that we are misusing something, which isn't necessarily true.

Unsure about kind vs k3s or alternatives, but this decision can be postponed until the general architecture is decided on.

I haven't looked into DRBD and pacemaker, but this feels like a whole new field that needs to be handled and learned by anyone in ermergency teams

[vknabel]

What about a semantic naming scheme? Like bootstrap cluster that creates the control cluster?

[majst01]

Great summary, though it feels a little bit complicated for the fact that we are trying to eat our own dog food. Just a feeling that we are misusing something, which isn't necessarily true.

Unsure about kind vs k3s or alternatives, but this decision can be postponed until the general architecture is decided on.

my actual feeling tends more to k3s TBH, but as you said, this decision can be done later

I haven't looked into DRBD and pacemaker, but this feels like a whole new field that needs to be handled and learned by anyone in ermergency teams

This is the case for any solution we want to introduce to make the needle HA. This is my first attempt for a quite simple one. I think we can even burn a ISO for the needle servers