Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The metric "opentelemetry_allocator_targets" is not accurate #3579

Open
chenlujjj opened this issue Dec 30, 2024 · 2 comments
Open

The metric "opentelemetry_allocator_targets" is not accurate #3579

chenlujjj opened this issue Dec 30, 2024 · 2 comments
Labels
bug Something isn't working needs triage

Comments

@chenlujjj
Copy link

chenlujjj commented Dec 30, 2024

Component(s)

target allocator

What happened?

Description

The metric "opentelemetry_allocator_targets" is not accurate

Steps to Reproduce

Deploy the target allocator within a k8s cluster in which there are otel-collectors and service monitors.

Expected Result

The metric "opentelemetry_allocator_targets" should reflect the accurate number of targets

Actual Result

Seems the metrics value is not equal to number of targets.

For example, in this cluster there is only one target, grafana-operator, but the metric value is 700+
image

As a contrast, according to metric opentelemetry_allocator_targets_per_collector, the total targets number is 1

image

Kubernetes Version

1.30.0

Operator version

v0.103.0

Collector version

v0.98.0

Environment information

No response

Log output

No response

Additional context

No response

@chenlujjj chenlujjj added bug Something isn't working needs triage labels Dec 30, 2024
@swiatekm
Copy link
Contributor

swiatekm commented Jan 2, 2025

This metric measures the number of targets discovered by the service discovery mechanism, before relabelling. Would that account for the discrepancy?

@chenlujjj
Copy link
Author

Let me double check for that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage
Projects
None yet
Development

No branches or pull requests

2 participants