Merge pull request #258 from kube-hetzner/add-cilium

Add cilium

README.md

@@ -8,7 +8,7 @@
 <br />
 <p align="center">
   <a href="https://github.com/mysticaltech/kube-hetzner">
-    <img src="https://github.com/kube-hetzner/kube-hetzner/raw/master/.images/kube-hetzner-logo.png" alt="Logo" width="112" height="112">
+    <img src="https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kube-hetzner-logo.png" alt="Logo" width="112" height="112">
   </a>
 
   <h2 align="center">Kube-Hetzner</h2>
@@ -39,7 +39,7 @@ _Please note that we are not affiliates of Hetzner; this is just an open-source
 - Ability to add nodes and nodepools when the cluster is running.
 - Traefik ingress controller attached to a Hetzner load balancer with proxy protocol turned on.
 - Possibility to turn Longhorn on, and optionally also turn Hetzner CSI off.
-- Ability to switch to Calico as CNI, and Cilium can also be easily added.
+- Ability to switch from Flannel to Calico or Cilium as CNI.
 - Tons of flexible configuration options to suit all needs.
 
 _It uses Terraform to deploy as it's easy to use, and Hetzner provides a great [Hetzner Terraform Provider](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs)._
@@ -68,7 +68,7 @@ brew install hcloud
 ### 💡 [Do not skip] Creating your kube.tf file
 
 1. Create a project in your [Hetzner Cloud Console](https://console.hetzner.cloud/), and go to **Security > API Tokens** of that project to grab the API key. Take note of the key! ✅
-2. Generate a passphrase-less ed25519 SSH key pair for your cluster; take note of the respective paths of your private and public keys. Or, see our detailed [SSH options](https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md). ✅
+2. Generate a passphrase-less ed25519 SSH key pair for your cluster; take note of the respective paths of your private and public keys. Or, see our detailed [SSH options](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/docs/ssh.md). ✅
 3. Prepare the module by copying `kube.tf.example` to `kube.tf` **in a new folder** which you cd into, then replace the values from steps 1 and 2. ✅
 4. (Optional) Many variables in `kube.tf` can be customized to suit your needs, you can do so if you want. ✅
 5. At this stage you should be in your new folder, with a fresh `kube.tf` file, if it is so, you can proceed forward! ✅
@@ -99,6 +99,12 @@ export KUBECONFIG=/<path-to>/clustername_kubeconfig.yaml
 
 _Once you start with Terraform, it's best not to change the state manually in Hetzner; otherwise, you'll get an error when you try to scale up or down or even destroy the cluster._
 
+## CNI
+
+The default is flannel, but you can also choose Calico or Cilium, by setting the cni_plugin variable in `kube.tf` to `calico` or `cilium`. For both, you can create a HelmChartConfig to edit their respective configuration post cluster deploy! Read more about HelmChartConfig [here](https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig).
+
+As Cilium has a lot of interesting and powerful configurations possibility. We give you the possibiliy to add a `cilium_values.yaml` file to the root of your module before you deploy your cluster, the same place where you have your `kube.tf` file. This file must be of the same format as the [Cilium values.yaml file](https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml), but with the values you want to modify. You can also find the default values that we use in the [cilium.yaml.tpl][https://github.com/kube-hetzner/kube-hetzner/blob/master/templates/cilium.yaml.tpl] file. During the deploy, Terraform will test to see if this file is present and if so will use those values to deploy the Cilium Helm chart.
+
 ### Scaling Nodes
 
 Two things can be scaled: the number of nodepools or the number of nodes in these nodepools. You have two lists of nodepools you can add to your `kube.tf`, the control plane nodepool and the agent nodepool list. Combined, they cannot exceed 255 nodepools (you are extremely unlikely to reach this limit). As for the count of nodes per nodepools, if you raise your limits in Hetzner, you can have up to 64,670 nodes per nodepool (also very unlikely to need that much).
@@ -125,7 +131,7 @@ By default, MicroOS gets upgraded automatically on each node and reboot safely v
 
 As for k3s, it also automatically upgrades thanks to Rancher's [system upgrade controller](https://github.com/rancher/system-upgrade-controller). By default, it follows the k3s `stable` channel, but you can also change to the `latest` one if needed or specify a target version to upgrade to via the upgrade plan.
 
-You can copy and modify the [one in the templates](https://github.com/kube-hetzner/kube-hetzner/blob/master/templates/plans.yaml.tpl) for that! More on the subject in [k3s upgrades](https://rancher.com/docs/k3s/latest/en/upgrades/basic/).
+You can copy and modify the [one in the templates](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/templates/plans.yaml.tpl) for that! More on the subject in [k3s upgrades](https://rancher.com/docs/k3s/latest/en/upgrades/basic/).
 
 ### Turning Off Automatic Upgrade
 
@@ -161,6 +167,31 @@ Rarely needed, but can be handy in the long run. During the installation, we aut
 
 <details>
 
+With Kube-Hetzner, you have the possibility to use Cilium as a CNI. It's very powerful and has great observability features. Below you will find a few useful commands.
+
+### Useful Cilium commands
+
+- Check the status of cilium with the following commands (get the cilium pod name first and replace it in the command):
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status --verbose
+```
+
+- Monitor cluster traffic with:
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium monitor
+```
+
+- See the list of kube services with:
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium service list
+```
+
+_For more cilium commands, please refer to their corresponding [Documentation](https://docs.cilium.io/en/latest/cheatsheet)._
+
 <summary>Ingress with TLS</summary>
 
 You have two solutions, the first is to use `Cert-Manager` to take care of the certificates, and the second is to let `Traefik` bear this responsability.
@@ -314,9 +345,9 @@ _Also, if you had a full-blown cluster in use, it would be best to delete the wh
 
 ## History
 
-This project has tried two other OS flavors before settling on MicroOS. Fedora Server, and k3OS. The latter, k3OS, is now defunct! However, our code base for it lives on in the [k3os branch](https://github.com/kube-hetzner/kube-hetzner/tree/k3os). Do not hesitate to check it out, it should still work.
+This project has tried two other OS flavors before settling on MicroOS. Fedora Server, and k3OS. The latter, k3OS, is now defunct! However, our code base for it lives on in the [k3os branch](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/k3os). Do not hesitate to check it out, it should still work.
 
-There is also a branch where openSUSE MicroOS came preinstalled with the k3s RPM from devel:kubic/k3s, but we moved away from that solution as the k3s version was rarely getting updates. See the [microOS-k3s-rpm](https://github.com/kube-hetzner/kube-hetzner/tree/microOS-k3s-rpm) branch for more.
+There is also a branch where openSUSE MicroOS came preinstalled with the k3s RPM from devel:kubic/k3s, but we moved away from that solution as the k3s version was rarely getting updates. See the [microOS-k3s-rpm](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/microOS-k3s-rpm) branch for more.
 
 ## Contributing
 
@@ -351,4 +382,4 @@ Code contributions are very much **welcome**.
 [issues-url]: https://github.com/mysticaltech/kube-hetzner/issues
 [license-shield]: https://img.shields.io/github/license/mysticaltech/kube-hetzner.svg?style=for-the-badge
 [license-url]: https://github.com/mysticaltech/kube-hetzner/blob/master/LICENSE.txt
-[product-screenshot]: https://github.com/kube-hetzner/kube-hetzner/raw/master/.images/kubectl-pod-all-17022022.png
+[product-screenshot]: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kubectl-pod-all-17022022.png

cilium_values.yaml.example

@@ -0,0 +1,30 @@
+# All Cilium helm values can be found at https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml
+# The following is a personal working and optimized example of Cilium helm values.
+
+ipam:
+  # -- Configure IP Address Management mode.
+  mode: kubernetes
+
+# -- Specify which network interfaces can run the eBPF datapath. This means
+# that a packet sent from a pod to a destination outside the cluster will be
+# masqueraded (to an output device IPv4 address), if the output device runs the
+# program. When not specified, probing will automatically detect devices.
+devices: "eth1"
+
+# -- Configure the encapsulation configuration for communication between nodes.
+# Possible values:
+#   - disabled (native routing works, however it feels that geneve is more stable, but we may be wrong)
+#   - vxlan
+#   - geneve (it's basically a better version of vxlan)
+tunnel: geneve
+
+# -- Configure Kubernetes specific configuration
+k8s:
+  # -- requireIPv4PodCIDR enables waiting for Kubernetes to provide the PodCIDR
+  # range via the Kubernetes node resource
+  requireIPv4PodCIDR: true
+
+# -- Configure the kube-proxy replacement in Cilium BPF datapath
+# Valid options are "disabled", "probe", "partial", "strict".
+# ref: https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/
+kubeProxyReplacement: strict

control_planes.tf

@@ -94,19 +94,20 @@ resource "null_resource" "control_planes" {
       token                       = random_password.k3s_token.result
       disable-cloud-controller    = true
       disable                     = local.disable_extras
-      flannel-iface               = "eth1"
       kubelet-arg                 = ["cloud-provider=external", "volume-plugin-dir=/var/lib/kubelet/volumeplugins"]
       kube-controller-manager-arg = "flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins"
+      flannel-iface               = "eth1"
       node-ip                     = module.control_planes[each.key].private_ipv4_address
       advertise-address           = module.control_planes[each.key].private_ipv4_address
       node-label                  = each.value.labels
       node-taint                  = each.value.taints
-      disable-network-policy      = var.cni_plugin == "calico" ? true : var.disable_network_policy
       write-kubeconfig-mode       = "0644" # needed for import into rancher
       },
-      var.cni_plugin == "calico" ? {
-        flannel-backend = "none"
-    } : {}))
+      lookup(local.cni_k3s_settings, var.cni_plugin, {}),
+      var.use_control_plane_lb ? {
+        tls-san = [hcloud_load_balancer.control_plane.*.ipv4[0], hcloud_load_balancer_network.control_plane.*.ip[0]]
+      } : {}
+    ))
 
     destination = "/tmp/config.yaml"
   }

init.tf

@@ -14,19 +14,18 @@ resource "null_resource" "first_control_plane" {
       cluster-init                = true
       disable-cloud-controller    = true
       disable                     = local.disable_extras
-      flannel-iface               = "eth1"
       kubelet-arg                 = ["cloud-provider=external", "volume-plugin-dir=/var/lib/kubelet/volumeplugins"]
       kube-controller-manager-arg = "flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins"
+      flannel-iface               = "eth1"
       node-ip                     = module.control_planes[keys(module.control_planes)[0]].private_ipv4_address
       advertise-address           = module.control_planes[keys(module.control_planes)[0]].private_ipv4_address
       node-taint                  = local.control_plane_nodes[keys(module.control_planes)[0]].taints
       node-label                  = local.control_plane_nodes[keys(module.control_planes)[0]].labels
-      disable-network-policy      = var.cni_plugin == "calico" ? true : var.disable_network_policy
       },
-      var.cni_plugin == "calico" ? {
-        flannel-backend = "none"
-      } : {},
-    var.use_control_plane_lb ? { tls-san = [hcloud_load_balancer.control_plane.*.ipv4[0], hcloud_load_balancer_network.control_plane.*.ip[0]] } : {}))
+      lookup(local.cni_k3s_settings, var.cni_plugin, {}),
+      var.use_control_plane_lb ? {
+        tls-san = [hcloud_load_balancer.control_plane.*.ipv4[0], hcloud_load_balancer_network.control_plane.*.ip[0]]
+    } : {}))
 
     destination = "/tmp/config.yaml"
   }
@@ -100,7 +99,7 @@ resource "null_resource" "kustomization" {
           "https://raw.githubusercontent.com/hetznercloud/csi-driver/${local.csi_version}/deploy/kubernetes/hcloud-csi.yml"
         ],
         var.traefik_enabled ? ["traefik_config.yaml"] : [],
-        var.cni_plugin == "calico" ? ["https://projectcalico.docs.tigera.io/manifests/calico.yaml"] : [],
+        lookup(local.cni_install_resources, var.cni_plugin, []),
         var.enable_longhorn ? ["longhorn.yaml"] : [],
         var.enable_cert_manager || var.enable_rancher ? ["cert_manager.yaml"] : [],
         var.enable_rancher ? ["rancher.yaml"] : [],
@@ -112,7 +111,7 @@ resource "null_resource" "kustomization" {
           file("${path.module}/kustomize/system-upgrade-controller.yaml"),
           "ccm.yaml",
         ],
-        var.cni_plugin == "calico" ? ["calico.yaml"] : []
+        lookup(local.cni_install_resource_patches, var.cni_plugin, [])
       )
     })
     destination = "/var/post_install/kustomization.yaml"
@@ -158,6 +157,16 @@ resource "null_resource" "kustomization" {
     destination = "/var/post_install/calico.yaml"
   }
 
+  # Upload the cilium install file
+  provisioner "file" {
+    content = templatefile(
+      "${path.module}/templates/cilium.yaml.tpl",
+      {
+        values = indent(4, trimspace(fileexists("cilium_values.yaml") ? file("cilium_values.yaml") : local.default_cilium_values))
+    })
+    destination = "/var/post_install/cilium.yaml"
+  }
+
   # Upload the system upgrade controller plans config
   provisioner "file" {
     content = templatefile(
@@ -226,7 +235,7 @@ resource "null_resource" "kustomization" {
       # Wait for k3s to become ready (we check one more time) because in some edge cases,
       # the cluster had become unvailable for a few seconds, at this very instant.
       <<-EOT
-      timeout 120 bash <<EOF
+      timeout 180 bash <<EOF
         until [[ "\$(kubectl get --raw='/readyz' 2> /dev/null)" == "ok" ]]; do
           echo "Waiting for the cluster to become ready..."
           sleep 2
@@ -238,7 +247,8 @@ resource "null_resource" "kustomization" {
       # Ready, set, go for the kustomization
       "kubectl apply -k /var/post_install",
       "echo 'Waiting for the system-upgrade-controller deployment to become available...'",
-      "kubectl -n system-upgrade wait --for=condition=available --timeout=120s deployment/system-upgrade-controller",
+      "kubectl -n system-upgrade wait --for=condition=available --timeout=180s deployment/system-upgrade-controller",
+      "sleep 5", # important as the system upgrade controller CRDs sometimes don't get ready right away, especially with Cilium.
       "kubectl -n system-upgrade apply -f /var/post_install/plans.yaml"
       ],
       local.using_klipper_lb || var.traefik_enabled == false ? [] : [

kube.tf.example

@@ -204,14 +204,14 @@ module "kube-hetzner" {
   # use_klipper_lb = "true"
 
   # If you want to configure a different CNI for k3s, use this flag
-  # possible values: flannel (Default), calico
-  # Cilium or other would be easy to add, you can mirror how Calico was added. PRs are welcome!
+  # possible values: flannel (Default), calico, and cilium
   # CAVEATS: Calico is not supported when not using the Hetzner LB (like when use_klipper_lb is set to true or when using a single node cluster),
   # because of the following issue https://github.com/k3s-io/klipper-lb/issues/6.
-  # cni_plugin = "calico"
+  # As for Cilium, we allow infinite configurations, please check the CNI section of the readme over at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/#cni.
+  # cni_plugin = "cilium"
 
   # If you want to disable the k3s default network policy controller, use this flag!
-  # Calico overrides this value to true automatically, the default is "false".
+  # Both Calico and Ciliun cni_plugin values override this value to true automatically, the default is "false".
   # disable_network_policy = true
 
   # If you want to disable the automatic use of placement group "spread". See https://docs.hetzner.com/cloud/placement-groups/overview/

locals.tf

@@ -23,8 +23,12 @@ locals {
 
   apply_k3s_selinux = ["/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp"]
 
-  install_k3s_server = concat(local.common_commands_install_k3s, ["curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=${var.initial_k3s_channel} INSTALL_K3S_EXEC=server sh -"], local.apply_k3s_selinux)
-  install_k3s_agent  = concat(local.common_commands_install_k3s, ["curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=${var.initial_k3s_channel} INSTALL_K3S_EXEC=agent sh -"], local.apply_k3s_selinux)
+  install_k3s_server = concat(local.common_commands_install_k3s, [
+    "curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=${var.initial_k3s_channel} INSTALL_K3S_EXEC=server sh -"
+  ], local.apply_k3s_selinux)
+  install_k3s_agent = concat(local.common_commands_install_k3s, [
+    "curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=${var.initial_k3s_channel} INSTALL_K3S_EXEC=agent sh -"
+  ], local.apply_k3s_selinux)
 
   control_plane_nodes = merge([
     for pool_index, nodepool_obj in var.control_plane_nodepools : {
@@ -69,7 +73,9 @@ locals {
   using_klipper_lb = var.use_klipper_lb || local.is_single_node_cluster
 
   # disable k3s extras
-  disable_extras = concat(["local-storage"], local.using_klipper_lb ? [] : ["servicelb"], var.traefik_enabled ? [] : ["traefik"], var.metrics_server_enabled ? [] : ["metrics-server"])
+  disable_extras = concat(["local-storage"], local.using_klipper_lb ? [] : ["servicelb"], var.traefik_enabled ? [] : [
+    "traefik"
+  ], var.metrics_server_enabled ? [] : ["metrics-server"])
 
   # Default k3s node labels
   default_agent_labels         = concat([], var.automatically_upgrade_k3s ? ["k3s_upgrade=true"] : [])
@@ -237,4 +243,36 @@ locals {
   labels_agent_node = {
     role = "agent_node"
   }
+
+  cni_install_resources = {
+    "calico" = ["https://projectcalico.docs.tigera.io/manifests/calico.yaml"]
+    "cilium" = ["cilium.yaml"]
+  }
+
+  cni_install_resource_patches = {
+    "calico" = ["calico.yaml"]
+  }
+
+  cni_k3s_settings = {
+    "flannel" = {
+      disable-network-policy = var.disable_network_policy
+    }
+    "calico" = {
+      disable-network-policy = true
+      flannel-backend        = "none"
+    }
+    "cilium" = {
+      disable-network-policy = true
+      flannel-backend        = "none"
+    }
+  }
+
+  default_cilium_values = <<EOT
+ipam:
+ operator:
+  clusterPoolIPv4PodCIDRList:
+   - ${local.cluster_cidr_ipv4}
+devices: "eth1"
+EOT
 }
+

templates/cilium.yaml.tpl

@@ -0,0 +1,12 @@
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: cilium
+  namespace: kube-system
+spec:
+  chart: cilium
+  repo: https://helm.cilium.io/
+  targetNamespace: kube-system
+  bootstrap: true
+  valuesContent: |-
+    ${values}