Failing to push logs to wavefront with Prometheus remote_write

[liviugavrila]

I am trying to push some prometheus logs to Wavefront using Prometheus's remote_write.

I am using a prometheus-storage-adapter and a wavefront proxy.

None of the PodMonitor logs get pushed. I can push Cadvisor logs with no problem.

I can see all the jvm.* logs in prometheus. But if I kubectl get --raw http://broker-endpoint, I cannot see any jvm type logs in the output.

Seems the logs get to the proxy, sometimes I get this timeout in the adapter.

Please advise.

I'm using Strimzi 0.21.1

Below my Prometheus config.
`global:
scrape_interval: 30s
scrape_timeout: 10s
evaluation_interval: 30s
external_labels:
prometheus: strimzi/strimzi-prometheus
prometheus_replica: prometheus-strimzi-prometheus-0
alerting:
alert_relabel_configs:

• separator: ;
  regex: prometheus_replica
  replacement: $1
  action: labeldrop
  alertmanagers:
• scheme: http
  path_prefix: /
  timeout: 10s
  api_version: v1
  relabel_configs:
  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: alertmanager
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: alertmanager
    replacement: $1
    action: keep
    kubernetes_sd_configs:
  • role: endpoints
    namespaces:
    names:
    • strimzi
      rule_files:
• /etc/prometheus/rules/prometheus-strimzi-prometheus-rulefiles-0/*.yaml
  scrape_configs:
• job_name: strimzi/kafka-bootstrap-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  • source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
    separator: ;
    regex: kafka-monitoring
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: tcp-prometheus
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  • source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: tcp-prometheus
    action: replace
    kubernetes_sd_configs:
  • role: endpoints
    namespaces:
    names:
    • strimzi
• job_name: strimzi/bridge-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  • source_labels: [__meta_kubernetes_pod_label_strimzi_io_kind]
    separator: ;
    regex: KafkaBridge
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: rest-api
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • separator: ;
    regex: (.*)
    target_label: job
    replacement: strimzi/bridge-metrics
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: rest-api
    action: replace
    kubernetes_sd_configs:
  • role: pod
    namespaces:
    names:
    • strimzi
• job_name: strimzi/cluster-operator-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  • source_labels: [__meta_kubernetes_pod_label_strimzi_io_kind]
    separator: ;
    regex: cluster-operator
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • separator: ;
    regex: (.*)
    target_label: job
    replacement: strimzi/cluster-operator-metrics
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
    kubernetes_sd_configs:
  • role: pod
    namespaces:
    names:
    • strimzi
• job_name: strimzi/entity-operator-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  • source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
    separator: ;
    regex: entity-operator
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: healthcheck
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • separator: ;
    regex: (.*)
    target_label: job
    replacement: strimzi/entity-operator-metrics
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: healthcheck
    action: replace
    kubernetes_sd_configs:
  • role: pod
    namespaces:
    names:
    • strimzi
• job_name: strimzi/kafka-resources-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  • source_labels: [__meta_kubernetes_pod_label_strimzi_io_kind]
    separator: ;
    regex: Kafka|KafkaConnect|KafkaConnectS2I|KafkaMirrorMaker|KafkaMirrorMaker2
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: tcp-prometheus
    replacement: $1
    action: keep
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  • separator: ;
    regex: (.*)
    target_label: job
    replacement: strimzi/kafka-resources-metrics
    action: replace
  • separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: tcp-prometheus
    action: replace
  • separator: ;
    regex: meta_kubernetes_pod_label(strimzi_io.+)
    replacement: $1
    action: labelmap
  • source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node_name
    replacement: $1
    action: replace
  • source_labels: [__meta_kubernetes_pod_host_ip]
    separator: ;
    regex: (.*)
    target_label: node_ip
    replacement: $1
    action: replace
    kubernetes_sd_configs:
  • role: pod
    namespaces:
    names:
    • strimzi
      remote_write:
• url: http://storage-adapter-service.strimzi.svc.cluster.local/receive
  remote_timeout: 30s
  write_relabel_configs:
  • source_labels: [name]
    separator: ;
    regex: jvm.*
    replacement: $1
    action: keep
    queue_config:
    capacity: 2500
    max_shards: 200
    min_shards: 1
    max_samples_per_send: 500
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 100ms
    metadata_config:
    send: true
    send_interval: 1m`

[scholzj]

I'm sorry, but I have no idea what Wavefront or Prometheus storage adapter are. I'm not sure I understand the relationship to Strimzi - we just provide a metrics endpoint which you can scrape with Prometheus, that is it.

I'm not sure if @sknot-rh has any idea, but maybe this question is better suited for some Prometheus or Wavefront forum?

[liviugavrila]

Thank you for the answer.

Just trying to understand how the PodMonitor metrics are different from the kubernetes-cadvisor metrics

For example if I run kubectl get --raw http://broker-endpoint , I can see no jvm* logs.
But they are present in Prometheus.

[scholzj]

TBH, I'm not sure what do you expect to get from kubectl get --raw http://broker-endpoint, sorry. What is http://broker-endpoint?

The jvm metrics in our endpoints are describing the state of the JVM (Java Virtual Machine) since Strimzi and Kafka are written in Java/Scala and run on top of the JVM. The metrics are important to monitor the state of the JVM.

[liviugavrila]

You can see all the metrics if you access the endpoints from the targets

If I access the metrics from cadvisor I can see the real metrics

If I access kafka metrics I see these metrics

But somehow in prometheus I can see the correct kafka metrics

[scholzj]

Well, for example the jvm_buffer_count_buffers you have on your screenshot comes from the operators and not from Kafka it self. You can exec into the topic-operator container (in the entity-operator pod) and do curl http://localhost:8080/metrics to see the actual metrics. I do not think using the kubectl get --raw command with the target endpoint from Prometheus does what you think it does ... it seems to return some metrics, but they have definitely nothing to do with Strimzi / Kafka.