The nginx_ingress_controller_requests metric not exposing the complete requestURI as label

[Laveshsp]

What happened:

I have installed nginx ingress controller and enabled controller metrics. Also added prometheus pod annotations to scrape metrics. When I use nginx_ingress_controller_requests metrics, it doesn't give me the entire requestURI in labels but just gives the host and path configured in kubernetes ingress manifest.

Query Ran :

rate(nginx_ingress_controller_requests{status=~"5.."}[1m]) > 0

Below are the labels that were output

{app_kubernetes_io_component="controller", app_kubernetes_io_instance="ingress-nginx", app_kubernetes_io_managed_by="Helm", app_kubernetes_io_name="ingress-nginx", app_kubernetes_io_part_of="ingress-nginx", app_kubernetes_io_version="1.9.3", cluster="SHARED-CUSTOMER-DEV-CL", controller_class="k8s.io/ingress-nginx", controller_namespace="ingress", controller_pod="ingress-nginx-controller-7cf96d6db6-5rrm4", helm_sh_chart="ingress-nginx-4.8.2", host="unique.py.lav.test", ingress="error-ingress", instance="10.135.69.73:10254", job="kubernetes-pods", kubernetes_namespace="ingress", kubernetes_pod_name="ingress-nginx-controller-7cf96d6db6-5rrm4", method="GET", namespace="monitor", path="/", pod_template_hash="7cf96d6db6", prometheus="prometheus/kube-prometheus-stack-prometheus", region="us-west-2", service="error-service", status="503"}

What you expected to happen:

I wanted the nginx_ingress_controller_requests to provide me the requestURI as label output which gives me the complete URI of the request

NGINX Ingress controller version : 1.9.3

Kubernetes version : 1.30

Environment:

• Cloud provider or hardware configuration: AWS EKS 1.30

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of ingress-nginx ingress 36 2024-11-04 11:53:23.847501744 +0000 UTC deployed ingress-nginx-4.8.2 1.9.3
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

USER-SUPPLIED VALUES:
clusterName: SHARED-CUSTOMER-DEV-CL
controller:
  allowSnippetAnnotations: true
  autoscaling:
    enabled: true
    maxReplicas: 2
  config:
    enable-modsecurity: true
    enable-real-ip: true
    enable-underscores-in-headers: true
    http-snippet: 'more_set_headers "X-ModSecurity: enabled";'
    log-format-escape-json: true
    log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
      "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
      $upstream_status, "upstreamAddr": "$upstream_addr", "httpRequest": {"requestMethod":
      "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
      $request_length, "responseSize": $upstream_response_length, "userAgent": "$http_user_agent",
      "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time
      s", "protocol": "$server_protocol"}}'
    modsecurity-snippet: |
      # this enables the default OWASP Core Rule Set
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

      # Enable prevention mode. Options: DetectionOnly On Off (default is DetectionOnly)
      SecRuleEngine On

      # Enable scanning of the request body
      SecRequestBodyAccess On

      # Reject if larger (we could also let it pass with ProcessPartial)
      SecRequestBodyLimitAction Reject

      # Send ModSecurity audit logs to the stdout (only for rejected requests)
      SecAuditLog /dev/stdout

      # Format the logs in JSON
      SecAuditLogFormat JSON

      # could be On/Off/RelevantOnly
      SecAuditEngine RelevantOnly
    use-forwarded-headers: true
    use-proxy-protocol: true
  extraArgs:
    default-ssl-certificate: ingress/default-tls
  metrics:
    enabled: true
  podAnnotations:
    '"prometheus.io/port"': 10254
    '"prometheus.io/scrape"': true
  replicaCount: 1
tolerations:
- effect: NoSchedule
  key: mainnode
  operator: Exists

Can you please let me know if there is an option to get the complete requestURI in mertics scraped by prometheus from nginx ingress controller

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[alphabet5]

This is expected behavior. If this were not the case, you'd have a cardinality explosion.

You should log your requests instead if you want the full request uri.

If you want metrics for specific paths, you can break them out in the ingress object, or use different ingress objects for different paths.

[longwuyuan]

/remove-kind bug

[Laveshsp]

@alphabet5 So if breaking the endpoints into different paths in ingress is not feasible at my end, then you are suggesting that the option left is to use log level monitoring for Prometheus to fetch the request uri during sending 4xx or 5xx alerts. With this, there will be additional log aggregator tools set up that might be needed i believe

So, was just curious if there is any way to configure nginx ingress controller to directly fetch request_uri as metrics or using any Prometheus exporters that can help me alert based on status code of requests at ingress level

[alphabet5]

Prometheus, or time-series metrics in any fashion are not appropriate for monitoring based on request_uri.

request_uri has high cardinality, which would create different metrics series in prometheus.

If you want statistics broken down by uri, you should either manually specify them by separating out different paths or hosts with your ingress configuration, or log the requests to something like elasticsearch where you can query request uri patterns.

You can use ingress-nginx metrics to alert on 4xx / 5xx requests, it's just not broken down by request_uri, just by ingress name, hostname, service, path, method, status, etc.

For example, if I were to break down prometheus metrics by request_uri, I'd have about 10,000,000 different series for the past day. This would make querying the metrics impossible, and the statistics around them useless, because each series would have so few requests that any insight into performance would be meaningless.

[Laveshsp]

@alphabet5 Thanks for detailed info and suggesting other possible options. Will look into exploring those 🙂

[longwuyuan]

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.