Prometheus addon crashes during start up

[webwurst]

Problem

Prometheus addon crashes during start up.

Logs

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/00-banner.sh
-----------------------------------------------------------
 Add-on: Prometheus
 Cloud native metrics
-----------------------------------------------------------
 Add-on version: fe262a9
 You are running the latest version of this add-on.
 System: Home Assistant OS 8.2  (aarch64 / raspberrypi4-64)
 Home Assistant Core: 2022.6.5
 Home Assistant Supervisor: 2022.05.3
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
cont-init: info: /etc/cont-init.d/00-banner.sh exited 0
cont-init: info: running /etc/cont-init.d/01-log-level.sh
cont-init: info: /etc/cont-init.d/01-log-level.sh exited 0
cont-init: info: running /etc/cont-init.d/prometheus.sh
cont-init: info: /etc/cont-init.d/prometheus.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun prometheus (no readiness notification)
services-up: info: copying legacy longrun prometheus-configgen (no readiness notification)
s6-rc: info: service legacy-services successfully started
[19:48:38] INFO: Starting prometheus...
[19:48:38] INFO: Starting prometheus config generator...
jq: error (at <stdin>:1): null (null) has no keys
ts=2022-06-14T17:48:39.218Z caller=main.go:491 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-06-14T17:48:39.219Z caller=main.go:535 level=info msg="Starting Prometheus Server" mode=server version="(version=2.36.1, branch=HEAD, revision=3c0a100dec2950f23e91f7efbf8867750960807d)"
ts=2022-06-14T17:48:39.219Z caller=main.go:540 level=info build_context="(go=go1.18.3, user=root@4827bbf13eda, date=20220609-15:14:08)"
ts=2022-06-14T17:48:39.219Z caller=main.go:541 level=info host_details="(Linux 5.15.32-v8 #1 SMP PREEMPT Thu Jun 9 01:50:59 UTC 2022 aarch64 40817795-prometheus local.hass.io)"
ts=2022-06-14T17:48:39.219Z caller=main.go:542 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-06-14T17:48:39.220Z caller=main.go:543 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-06-14T17:48:39.220Z caller=query_logger.go:90 level=error component=activeQueryTracker msg="Error opening query log file" file=/data/prometheus/queries.active err="open /data/prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7fe8adcc31, 0x10}, 0x14, {0x32e2bb8, 0x4000a144b0})
	/app/promql/query_logger.go:120 +0x2f4
main.main()
	/app/cmd/prometheus/main.go:597 +0x6108
[19:48:39] WARNING: Prometheus crashed, halting add-on
s6-rc: info: service legacy-services: stopping
[19:48:39] INFO: Prometheus config stopped, restarting...
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Additional info

I think the logs contains all the interesting information. I can add more if needed. This also happens after restarting Home Assistant.

[webwurst]

Maybe it's something around these changes?

• https://github.com/hassio-addons/addon-prometheus/blob/c64c377256e3b2749f5f98ef6f41224ca298c1a9/prometheus/rootfs/etc/cont-init.d/prometheus.sh
• https://github.com/hassio-addons/addon-prometheus/blob/844a3a4985a3b968d45d6608afeea54509407342/prometheus/rootfs/etc/fix-attrs.d/01-prometheus

[webwurst]

Is it possible to install a former release of this addon to verify it's working fine?

[42Network]

Hi @webwurst. I just submitted a PR that allows this EDGE addon to start without issues. It seems to all be working, with both Prometheus and HA metrics showing up. Plus Grafana can see the Prometheus data, too. You can grab the files from my github or just edit the few files by hand.

Now I'll have to decide whether to use Prometheus or InfluxBD for long-term metric storage...

[webwurst]

@42Network is there an easy way for me to test your container image? Like editing some config file with registry/container-name?

[webwurst]

Nice to see an upgrade to Prometheus version 2.40.1 :) Just the permission errors are still persisting and prevent Prometheus to start for me.

[amaciuc]

Any news about this issue? Prometheus add-on is still not working.

[PolestarWx]

Same for me

[lupino3]

Noting that in the review for #49, @frenck mentioned that prometheus should run as root: #49 (comment)