Strange behaviour after auto updates

[Breee]

Greetings,

I have run into two issues when the automatic updates hit and I hope that you guys can help me :)

1. I use a custom values.yaml for my traefik which I deploy using gitlab CI:

additionalArguments:
- --entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8
- --entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8
- --entryPoints.web.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8
- --entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8
global:
  systemDefaultRegistry: ""
globalArguments: []
image:
  name: rancher/mirrored-library-traefik
  tag: 2.9.4
podAnnotations:
  prometheus.io/port: "8082"
  prometheus.io/scrape: "true"
priorityClassName: system-cluster-critical
providers:
  kubernetesIngress:
    publishedService:
      enabled: true
service:
  annotations:
    load-balancer.hetzner.cloud/disable-private-ingress: "true"
    load-balancer.hetzner.cloud/ipv6-disabled: "true"
    load-balancer.hetzner.cloud/location: fsn1
    load-balancer.hetzner.cloud/name: <LB_NAME>
    load-balancer.hetzner.cloud/type: lb11
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    external-dns.alpha.kubernetes.io/hostname: <CLUSTER_DNS_NAME>
    external-dns.alpha.kubernetes.io/ttl: "120"
  enabled: true
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv4
  type: LoadBalancer
tolerations:
- key: CriticalAddonsOnly
  operator: Exists
- effect: NoSchedule
  key: node-role.kubernetes.io/control-plane
  operator: Exists
- effect: NoSchedule
  key: node-role.kubernetes.io/master
  operator: Exists

  helm repo add traefik https://helm.traefik.io/traefik
  helm repo update
  helm upgrade --install --namespace=kube-system traefik traefik/traefik --values values.yaml

The purpose is to use external DNS to automatically set the DNS entry of my cluster to my traefik LB IP.

After Automatic updates happen, this configuration is gone and my traefik LB gets IPv4 and IPv6 Ips as well as missing annotations in the service

2. Sometimes i have these volume errors after restarts:

| |  | 2022-12-19 09:46:13 | name=wiki-wikijs-867f4dcd48-dhn9d kind=Pod objectAPIversion=v1 objectRV=3470276 eventRV=3471466 sourcecomponent=kubelet sourcehost=phobos-worker-hel1-kfk reason=FailedMount type=Warning count=1 msg="Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[tz-config volume kube-api-access-dl5v9]: timed out waiting for the condition" | |
|--- | --- | --- | --- | ---|
| |  | 2022-12-19 09:43:56 | name=wiki-wikijs-867f4dcd48-dhn9d kind=Pod objectAPIversion=v1 objectRV=3470276 eventRV=3470870 sourcecomponent=kubelet sourcehost=phobos-worker-hel1-kfk reason=FailedMount type=Warning count=1 msg="Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[kube-api-access-dl5v9 tz-config volume]: timed out waiting for the condition" | |
| |  | 2022-12-19 09:41:53 | name=wiki-wikijs-867f4dcd48-dhn9d kind=Pod action=Binding objectAPIversion=v1 objectRV=3470273 eventRV=3470279 reportinginstance=default-scheduler-phobos-control-plane-hel1-pmu reportingcontroller=default-scheduler reason=Scheduled type=Normal msg="Successfully assigned wiki/wiki-wikijs-867f4dcd48-dhn9d to phobos-worker-hel1-kfk" | |
| |  | 2022-12-19 09:41:53 | name=wiki-wikijs-867f4dcd48-dhn9d kind=Pod objectAPIversion=v1 objectRV=3470276 eventRV=3470286 sourcecomponent=attachdetach-controller reason=FailedAttachVolume type=Warning count=1 msg="Multi-Attach error for volume \"wiki-volume-pv\" Volume is already used by pod(s) wiki-db-66f746d46f-zzsrh" | |

I'm using longhorn with the following PV / PVC:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"wiki-volume-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"300Mi"},"csi":{"driver":"driver.longhorn.io","fsType":"ext4","volumeAttributes":{"numberOfReplicas":"3","staleReplicaTimeout":"2880"},"volumeHandle":"wiki-volume"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"longhorn","volumeMode":"Filesystem"}}
    longhorn.io/volume-scheduling-error: ""
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2022-12-23T08:16:00Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/driver-longhorn-io
  name: wiki-volume-pv
  resourceVersion: "5027482"
  uid: 0a9a4db4-b85c-4e2a-8665-03bf372393ef
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 300Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: wiki-volume
    namespace: wiki
    resourceVersion: "5027362"
    uid: e81e524d-46e9-4258-8998-3986f8eb0567
  csi:
    driver: driver.longhorn.io
    fsType: ext4
    volumeAttributes:
      numberOfReplicas: "3"
      staleReplicaTimeout: "2880"
    volumeHandle: wiki-volume
  persistentVolumeReclaimPolicy: Delete
  storageClassName: longhorn
  volumeMode: Filesystem
status:
  phase: Bound

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"wiki-volume","namespace":"wiki"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"300Mi"}},"storageClassName":"longhorn","volumeName":"wiki-volume-pv"}}
    pv.kubernetes.io/bind-completed: "yes"
  creationTimestamp: "2022-12-23T08:16:01Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: wiki-volume
  namespace: wiki
  resourceVersion: "5027477"
  uid: e81e524d-46e9-4258-8998-3986f8eb0567
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 300Mi
  storageClassName: longhorn
  volumeMode: Filesystem
  volumeName: wiki-volume-pv
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 300Mi
  phase: Bound

[mysticaltech]

@Breee Your Traefik values.yaml content looks fine, but you better include those in the corresponding section in the kube.tf file, and not re-install in via helm. This is done automatically.

About the Longhorn errors, seeing some errors are ok if at the end it resolves itself on its own. Does it do so?

[Breee]

@mysticaltech Yeah, in the meantime i figured out that the configs are stored in the helmchartconfigs.helm.cattle.io ressources since the helm-controller ist used. Does terraform patch the the helmchartconfigs if I change my kube.tf?

about longhorn: the error did not appear anymore, so i'm not sure what happend that day. I had to get rid of the PV and PVC tho and redeploy.

[mysticaltech]

Yes, exactly we deploy a helmchartconfig to give traefik the required values.

[Breee]

Yep

I did not figure out how set the values.yaml via kube.tf

I only see this: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/kube.tf.example#L238
Which does not cover my usecase of adding additional annotations to the service

[Breee]

@mysticaltech I came across longhorn issues again today:

24s         Warning   FailedAttachVolume       pod/sonarqube-db-54d5d7f746-d5987        Multi-Attach error for volume "sonarqube-volume-pv" Volume is already exclusively attached to one node and can't be attached to another
7s          Normal    SuccessfulAttachVolume   pod/sonarqube-db-54d5d7f746-d5987        AttachVolume.Attach succeeded for volume "sonarqube-volume-pv"
4m37s       Warning   FailedMount              pod/sonarqube-db-54d5d7f746-qzdbl        Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[tz-config volume kube-api-access-8wwtp]: timed out waiting for the condition
22m         Warning   FailedMount              pod/sonarqube-db-54d5d7f746-qzdbl        Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[kube-api-access-8wwtp tz-config volume]: timed out waiting for the condition
9m8s        Warning   FailedMount              pod/sonarqube-db-54d5d7f746-qzdbl        Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[volume kube-api-access-8wwtp tz-config]: timed out waiting for the condition

I had to delete the PV + PVC and redeploy
Seems like there was a dangling PV/PVC after the reboot of a node ?
How do you guys avoid this?

Using ReadWriteMany volumes could probably help, but they are pretty slow I think

[mysticaltech]

@kube-hetzner/core Any ideas why that would happen?

[WolfspiritM]

For the first issue:

When a master node boots it executes all manifests stored in:
/var/lib/rancher/k3s/server/manifests

Also the traefik helmchart is added automaticly as long as the k3s config doesn't include the option to remove traefik.

Instead of redeploying traefik completly or removing the traefik config the best way would be to just deploy a single HelmChartConfig to that folder as described here:
https://docs.k3s.io/networking#traefik-ingress-controller

Afaik this terraform module doesn't change any config if you change things in the kube.tf file and redeploy. It only applies the config once on first setup. Once the files have been created they stay as they were on first setup.

For the longhorn issue:
I sadly can't help with that. I'm having issues with Longhorn myself (corrupted FS and so on) which is really annoying and one of the reasons I might move away from self managed Kubernetes in the near future...

[mysticaltech]

@WolfspiritM For Longhorn did you check out the resource created by @ifeulner, maybe this could help you out. Here it is https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b