Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow to treat kubelet IPs as internal IPs when using robot nodes #851

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

codablock
Copy link

This introduces a new environment variable
(ROBOT_PROVIDED_NODE_IP_INTERNAL) which causes HCC to treat IPs give via
kubelet's --node-ip flag as internal node IP. The assumption here is
that robot nodes will always treat IP reported via the robot webservice
as external, because robot nodes don't know the concept of private IPs
directly. The only way to introduce private IPs is via other means, e.g.
vSwitches or VPNs. In that case, operators would usually provide the
internal IP to kubelet (via --node-ip), so that the kubelet only listens
on the internal IP instead of the public IP.

Kubelet provides this IP to the k8s Node object via the
"alpha.kubernetes.io/provided-node-ip" annotation, but only when the
cloud provider is set to external (which is of course the case for us).
We can leverage this annotation in HCC and use it to fill the
status.addresses field with this IP (in addition to the already added
NodeExternalIP and NodeHostnName).

This commit/PR also changes the load balancer provider to pass this same
IP to the target configuration in case ROBOT_PROVIDED_NODE_IP_INTERNAL
is enabled and "load-balancer.hetzner.cloud/use-private-ip=true" is
passed to the LoadBalancer SVC.

With these changes a robot node can go fully private and close down all
public ports via the robot firewall or other means. It essentially can
be treated like a private cloud node.

Fixes #676 #790 #760 #708 #635 #703 #571 #620
Replaces #696 (as it doesn't require static configuration of IPs at controller installation time)

This introduces a new environment variable
(ROBOT_PROVIDED_NODE_IP_INTERNAL) which causes HCC to treat IPs give via
kubelet's --node-ip flag as internal node IP. The assumption here is
that robot nodes will always treat IP reported via the robot webservice
as external, because robot nodes don't know the concept of private IPs
directly. The only way to introduce private IPs is via other means, e.g.
vSwitches or VPNs. In that case, operators would usually provide the
internal IP to kubelet (via --node-ip), so that the kubelet only listens
on the internal IP instead of the public IP.

Kubelet provides this IP to the k8s Node object via the
"alpha.kubernetes.io/provided-node-ip" annotation, but only when the
cloud provider is set to external (which is of course the case for us).
We can leverage this annotation in HCC and use it to fill the
status.addresses field with this IP (in addition to the already added
NodeExternalIP and NodeHostnName).

This commit/PR also changes the load balancer provider to pass this same
IP to the target configuration in case ROBOT_PROVIDED_NODE_IP_INTERNAL
is enabled and "load-balancer.hetzner.cloud/use-private-ip=true" is
passed to the LoadBalancer SVC.

With these changes a robot node can go fully private and close down all
public ports via the robot firewall or other means. It essentially can
be treated like a private cloud node.
@codablock codablock requested a review from a team as a code owner January 23, 2025 08:41
@codablock
Copy link
Author

FYI, I linked many issue that I believe all get fixed by this PR. The error many of the issues report is about the node controller not being able to identify the node via it's internal IP. This is because the node controller uses the status.addresses of the Node to identify the node, which is only known by its internal IP when using --node-ip on the kubelet.

This PR is also a good replacement for #696 IMHO as it requires much less maintenance by the cluster admin (no need to manually update the IP map when adding/removing/replacing nodes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Local IPs with Robots server on vSwitch not working
1 participant