Heathcheck problems after policy applied

[hazkall]

Hello!

Im having issues with Pods Heathcheck. Im using EKS v1.21.5-eks-9017834 and Linkerd stable-2.11.1.

And follow the documentation on https://buoyant.io/2021/12/14/locking-down-network-traffic-between-kubernetes-namespaces/ to lock my namespace.

Im apply the follow rule on my thanos config

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: thanos
  name: grpc-thanos
  labels:
    type: server-auth
spec:
  podSelector: 
    matchLabels: {}
  port: grpc
  proxyProtocol: gRPC

---

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: thanos
  name: http-thanos
  labels:
    type: server-auth
spec:
  podSelector: 
    matchLabels: {}
  port: http
  proxyProtocol: HTTP/1

---

apiVersion: policy.linkerd.io/v1beta1
kind: ServerAuthorization
metadata:
  namespace: thanos
  name: thanos-policy-namespaced
spec:
  server:
    selector:
      matchLabels:
        type: server-auth
  client:
    meshTLS:
      identities:
      - "*.thanos.serviceaccount.identity.linkerd.cluster.local"
      - "*.prometheus.serviceaccount.identity.linkerd.cluster.local"

And for the kubelet healthcheck im apply the follow config on the same namespace

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: thanos
  name: servicemesh-admin
spec:
  port: linkerd-admin
  podSelector:
    matchLabels: {}
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1beta1
kind: ServerAuthorization
metadata:
  namespace: thanos
  name: servicemesh-admin
spec:
  server:
    name: servicemesh-admin
  client:
    unauthenticated: true

On the events of namespace i have de follow error on Readness probe and Liveness probe

8m5s        Warning   Unhealthy               pod/thanos-query-66497f7745-r86z5             Readiness probe failed: HTTP probe failed with statuscode: 403
8m5s        Warning   Unhealthy               pod/thanos-query-frontend-5b854dfb65-xpp7h    Liveness probe failed: HTTP probe failed with statuscode: 403
5m58s       Warning   Unhealthy               pod/thanos-query-frontend-647d669d7d-zflcz    Readiness probe failed: HTTP probe failed with statuscode: 403
3m8s        Warning   BackOff                 pod/thanos-query-66497f7745-r86z5             Back-off restarting failed container
3m4s        Warning   BackOff                 pod/thanos-query-frontend-5b854dfb65-xpp7h    Back-off restarting failed container

If i disable the Readiness and Liveness probe on deployment, all works good

Thanos Query deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: thanos
    meta.helm.sh/release-namespace: thanos
  creationTimestamp: "2022-02-08T17:45:27Z"
  generation: 2
  labels:
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: thanos
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: thanos
    helm.sh/chart: thanos-9.0.2
  name: thanos-query
  namespace: thanos
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: query
      app.kubernetes.io/instance: thanos
      app.kubernetes.io/name: thanos
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: query
        app.kubernetes.io/instance: thanos
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: thanos
        helm.sh/chart: thanos-9.0.2
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/component: query
                  app.kubernetes.io/instance: thanos
                  app.kubernetes.io/name: thanos
              namespaces:
              - thanos
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - query
        - --log.level=info
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --query.replica-label=replica
        image: docker.io/bitnami/thanos:0.24.0-scratch-r0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /-/healthy
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        name: query
        ports:
        - containerPort: 10902
          name: http
          protocol: TCP
        - containerPort: 10901
          name: grpc
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /-/ready
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: false
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: thanos-query
      serviceAccountName: thanos-query
      terminationGracePeriodSeconds: 30

Thanos Query Logs inside Linkerd Proxy

[   787.888941s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36692
[   787.888961s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36692}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   797.888763s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36764
[   797.888802s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36764}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   797.888900s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36766
[   797.888914s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36766}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   807.889340s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36866
[   807.889377s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36866}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   807.889473s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36862
[   807.889494s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36862}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   817.888833s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36990
[   817.888877s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36990}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos
[   817.888977s]  INFO ThreadId(01) inbound:server{port=10902}: linkerd_app_inbound::policy::authorize::http: Request denied server=http-thanos tls=None(NoClientHello) client=10.11.11.239:36984
[   817.888997s]  INFO ThreadId(01) inbound:server{port=10902}:rescue{client.addr=10.11.11.239:36984}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server http-thanos


linkerd authz -n thanos deploy                                                                                                                                                          ✔ 
      SERVER                    AUTHORIZATION  
      servicemesh-admin         servicemesh-admin  
      grpc-thanos               thanos-policy-namespaced  
      http-thanos               thanos-policy-namespaced  

10.11.11.239 is my node IP, probable is the Kubelet

[vishnukv64]

I'm also facing health check issues with pods after injecting proxy and applying server authorization policies.

There is no proper documentation for deployments with health checks, all their demos are regarding emojivoto without health checks.

[mateiidavid]

The resource quoted above had an example for the admin server: https://buoyant.io/2021/12/14/locking-down-network-traffic-between-kubernetes-namespaces/.

cat << EOF | kubectl apply -f -
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: emojivoto
  name: admin
spec:
  port: linkerd-admin
  podSelector:
    matchLabels: {} # every pod
  proxyProtocol: HTTP/1
---
apiVersion: policy.linkerd.io/v1beta1
kind: ServerAuthorization
metadata:
  namespace: emojivoto
  name: admin-everyone
spec:
  server:
    name: admin
  client:
    unauthenticated: true
EOF

[mateiidavid]

@lipenodias hmm that looks a bit odd, as far as I can tell the authz resources look good. Is your thanos workload meshed? I know it's an obvious question but I didn't see any inject annotation in the template, so just making sure.

I think it'd be helpful to perhaps put linkerd=debug,warn as a log level and see what's happening, might give us a few more details about what's going on with the policy discovery mechanism.