What to do when a Control Panel Node died?

[jimping]

Hey there,
Since yesterday I am struggling with a problem I am not able to solve. I set up a default cluster just like the example where my first control panel node died. Not accessible via SSH either. Hetzner said hardware is ok. Now I am completely lost as I am only aware of kubernetes basics. Rescue system does not help me either.

Is there any easy solution how to fix that? I am not able do deploy or manage the cluster via kubectl anymore. Changing the Kube config IP to one of the other control panel does not work either. Not accessible from all entries I tried.

[WebSpider]

Do you have any clues when you open the console of the control-plane in hetzner's console.hetzner.cloud?

[jimping]

Not really. When I open the console I get a login prompt. As I do not have a root password (ssh-key) I am not able to login. Booting in rescue does not help either. I am unable to change the root password. I am getting the notice that the file system is read only. Even with mount -o rw/dev/sda3 /mnt I am unable to get write permissions. Hetzner support says that the hardware is healthy.

[WebSpider]

That's indeed exactly what I would do next .. Go to the rescue system and mount the filesystem on the rescueOS....
Note there is a small typo in your mount command:

mount /dev/sdaX /mnt -o rw where X is the id of your root filesystem, as scanned via lsblk