Prometheus extension adding underscores to datasource names in some metrics

[sixtus]

The prometheus extension is reporting the dataSource wrapped in underscores in some metrics, while as in in others it's reported as is.

Affected Version

I noticed this in Druid 31, it must have been introduced as a regression earlier, as this used to work.

Description

I am using prometheus extension to source the following grafana alert:

max_over_time(druid_ingest_kafka_lag{dataSource!~".*test.*"}[3m]) / on (dataSource) sum(rate(druid_ingest_events_processed_total{dataSource!~".*test.*"}[3m])) by (dataSource) > 60*15

In words: Alert if the ingestion lag is more than 15min based on the reported consumption speed. Like I said, this used to work just fine.

However:

druid_ingest_events_processed_total now reports dataSource with underscores, i.e. "_foo_" instead of "foo", while druid_ingest_kafka_lag still reports the dataSource as "foo". Thus they don't match anymore and my alert when silent.

Expected behavior

All prometheus metrics should report the dataSource as is and not wrap them in underscores. And if that's a feature, at least be consistent.

[sixtus]

I was able to circle it in a bit.

It seems to affect all (but only) metrics starting with "ingest" and tracing a bit through the git changes, I came to

https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/TaskRealtimeMetricsMonitorBuilder.java#L45

being added 7 months ago, just in time to qualify as the source of my issue.

@clintropolis any thoughts?

[ashwintumma23]

@sixtus,

• For the logging emitter, I am able to see the dataSource name correctly emitted, without any underscore (_) decorations.

• For Prometheus emitter, quick callout for reformatting of metric names and labels: https://druid.apache.org/docs/latest/development/extensions-contrib/prometheus/#metric-names

  Code References:

  • https://github.com/apache/druid/blob/master/extensions-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/Metrics.java#L90
  • https://github.com/apache/druid/blob/master/extensions-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java#L154

• Despite that being said, even I am curious why would the dataSource name be decorated with underscores only for a subset of metrics. Maybe, try running the unit tests (https://github.com/apache/druid/blob/master/extensions-contrib/prometheus-emitter/src/test/java/org/apache/druid/emitter/prometheus/PrometheusEmitterTest.java) with your data source configurations, and see if the same behavior is seen.