Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP get service and pod metrics via our prometheus scraper #204

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

WIP get service and pod metrics via our prometheus scraper #204

wants to merge 1 commit into from

Conversation

kwmonroe
Copy link
Contributor

This isn't quite ready yet, but it brings in the latest service and pod metrics functionality from the upstream prometheus scraper:

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

Since our prometheus is not in-cluster, we have to get svc/pod metrics via the load balancer. I lifted relabeling configs to do this from the following gist:

https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450

It's still not quite right though:

screen shot 2018-07-12 at 4 27 42 pm

As an example into the red, see what prometheus is effectively trying to do for the nginx-ingress pod:

$ curl -u admin:PASSWORD -k https://KUBEAPI_LOAD_BALANCER:443/api/v1/namespaces/default/pods/nginx-ingress-kubernetes-worker-controller-jp962:443/proxy/metrics
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.13.12</center>
</body>
</html>

You might say, "oh hey, just stop trying to hit pod:443 and you'll get past this". I'd be like, "OK", and then show you this:

$ curl -u admin:bm0fGt3hFNudiYc8GYbCIg8KsEURSME3 -k https://172.31.23.115:443/api/v1/namespaces/default/pods/nginx-ingress-kubernetes-worker-controller-jp962/proxy/metrics
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>

We do have some green, which makes me think we're on the right track. It could be some of these things are not producing metrics, or we're looking for them in the wrong place (maybe not proxied how we expect). I wanted to put this up to solicit ideas from anyone who may know how to scrape pods/services from a prometheus that is not in-cluster.

NOTE: to even get what we have now, we have to annotate pods and services so that the scraper sees them from the api server:

kubectl annotate pod --namespace='[default|kube-system]' --all prometheus.io/scrape=true
kubectl annotate svc --namespace='[default|kube-system]' --all prometheus.io/scrape=true

@adam-stokes adam-stokes changed the title [WIP] get service and pod metrics via our prometheus scraper WIP get service and pod metrics via our prometheus scraper Jul 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kwmonroe