RBAC Permission Failures on Existing Cluster, replicasets and nodes?

[jordanabakerafs]

Came back from a the weekend today and we're seeing a chunk of errors in our ingresscontroller, which is no longer processing traffic. This isn't all of them, but they're all looking for replicaset and node rbac access.

EX:

• Error collecting telemetry data: Replicas: replicasets.apps "nginx-ingress-controller-86684cd77b" is forbidden: User "system:serviceaccount:nginx-ingress:nginx-ingress" cannot get resource "replicasets" in API group "apps" in the namespace "nginx-ingress"
• Error collecting telemetry data: error generating InstallationID: replicasets.apps "nginx-ingress-controller-86684cd77b" is forbidden: User "system:serviceaccount:nginx-ingress:nginx-ingress" cannot get resource "replicasets" in API group "apps" in the namespace "nginx-ingress"
• E0624 21:16:42.625628 1 collector.go:159] Error collecting telemetry data: Nodes: nodes is forbidden: User "system:serviceaccount:nginx-ingress:nginx-ingress" cannot list resource "nodes" in API group "" at the cluster scope

We've got this same deployment mapped across eight environments but this is the only one failing, and after looking all day, we're at a loss.

We can see in the RBAC yaml in the repo that those permissions aren't deployed with the cluster, and as far as we know they aren't (shouldn't be?) required.

Stack is Azure Kubernetes, local accounts with Kubernetes RBAC.

As far as we can tell from our logs, nothing changed. We've redeployed the controller a few times but have been met with no success.

Does anyone in the community have any ideas here that might point us in the right direction?

[jjngx]

Hi @jordanabakerafs, you can opt-out from using telemetry. NGINX Ingress Controller docs provide information how to do it. In the NIC v3.6.0 (release date June 25th) permission (RBAC) issues won't be reported as errors.

cc / @shaun-nx

[jordanabakerafs]

Thanks for the suggestion @jjngx, we'll give that a go and report back.

[j1m-ryan]

Just noticed we need to update this argument in the docs from .telemetry to .telemetryReporting But should be possible to find in the values.yaml file
controller.telemetryReporting.enable=false

[jordanabakerafs]

Thanks for the info @jjngx and @j1m-ryan - Looks to have gotten us patched up.

@j1m-ryan - Does telemetry require node and replicaset rbac access? If so since it's enabled by default (according to the docs), should those also be included in the default clusterrolebinding template?

[j1m-ryan]

I believe they have been. list nodes was added here and get replicasets was added here. I suspect what happened in your case was you were using an older ClusterRole definition, with a more recent version of NIC. What was your upgrade process?

[jordanabakerafs]

Hi @j1m-ryan Good catch, and explains the behavior. We're pinned to an older version currently, and were pulling using OCI into an azure container registry and then pushing to a separate registry, so it looks like there may have been some desyncing occuring.

We updated everything based on your comment and everything appears to be working properly now.

I went ahead and marked answer and I'll close this one out.

Thanks for the assist @j1m-ryan and @jjngx !

[j1m-ryan]

Glad to hear you got it sorted!