error message every 10 minutes about managing the server certificate

[fredcy]

I'm getting error output like this every 10 minutes from acme-dns. Is this spurious? Can I do anything to prevent the error?

error        maintenance        unable to get configuration to manage certificate; unable to renew        {"identifiers": ["auth.xxxx.com"], "error": "config returned for certificate [auth.xxxx.com] is not nil and points to different cache; got 0xc000027b90, expected 0xc000027c70 (this one)"}

The certificate used by acme-dns for its HTTPS traffic works fine, in that there is no complaint from the client side. In config.cfg I have tls = "letsencrypt".

I'm running acme-dns as a systemd service, running as the unprivileged acme-dns user. /var/lib/acme-dns/api-certs and everything comprised in there is owned and writable by the acme-dns user.

I tried removing all of /var/lib/acme-dns/api-certs and running acme-dns service from scratch. It rebuilds the cert in a new /var/lib/acme-dns/api-certs directory, but I soon get the same recurring error messages.

It appears that the error message comes from the github.com/caddyserver/certmagic module used by acme-dns, but I have not been able to work out why it throws that error.

I ran acme-dns as root for a while before improving the setup to run as an unprivileged user. I wonder if that left some remnant that needs to be cleaned up, but I can't find any such thing.

[danielztolnai]

I am receiving the same messages. Did you manage to find out the cause?

I've never run acme-dns as root, so that can be ruled out. I built the executable from the latest source using go 1.18.1 and am running it on a fresh Ubuntu 22.04.3 using the provided systemd service. I also have tls = "letsencrypt" and I'm also using the recommended user setup.

[fredcy]

I ended up not using acme-dns and so I don't have more info.

[PKizzle]

May I ask what you are using as an alternative?

[fredcy]

(Probably not helpful, but...) I was planning to use acme-dns to manage certs on a private development network on a DNS sub-domain. It worked OK, but management decided to just buy a wildcard cert for that subdomain, making acme-dns moot in our case.

[PKizzle]

Ah okay. I thought there might be a different solution to acme-dns but that does not seem to be the case then.

[PKizzle]

For the rest that are facing the cache issue: I have found a solution but am not sure whether this is the correct patch as I have added quite a bit of source code to acme-dns. So try it out and give feedback whether it works for you. It is based on the refactoring branch.

From 003a56d677fe0cf621ea92fc9446cf45a199e277 Mon Sep 17 00:00:00 2001
From: Philipp Kolberg <[email protected]>
Date: Wed, 29 Nov 2023 22:43:49 +0100
Subject: [PATCH] Fix certmagic cache handling

---
 pkg/api/api.go | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/pkg/api/api.go b/pkg/api/api.go
index 9fc96f6..bd75129 100644
--- a/pkg/api/api.go
+++ b/pkg/api/api.go
@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"net/http"
+	"sync"
 
 	"github.com/acme-dns/acme-dns/pkg/acmedns"
 
@@ -14,10 +15,12 @@ import (
 )
 
 type AcmednsAPI struct {
-	Config  *acmedns.AcmeDnsConfig
-	DB      acmedns.AcmednsDB
-	Logger  *zap.SugaredLogger
-	errChan chan error
+	Config  	 *acmedns.AcmeDnsConfig
+	DB      	 acmedns.AcmednsDB
+	Logger  	 *zap.SugaredLogger
+	errChan 	 chan error
+	magicCache   *certmagic.Cache
+	magicCacheMu sync.Mutex
 }
 
 func Init(config *acmedns.AcmeDnsConfig, db acmedns.AcmednsDB, logger *zap.SugaredLogger, errChan chan error) AcmednsAPI {
@@ -137,12 +140,17 @@ func (a *AcmednsAPI) setupTLS(dnsservers []acmedns.AcmednsNS) *certmagic.Config
 	magicConf.Logger = a.Logger.Desugar()
 	magicConf.Storage = &storage
 	magicConf.DefaultServerName = a.Config.General.Domain
-	magicCache := certmagic.NewCache(certmagic.CacheOptions{
-		GetConfigForCert: func(cert certmagic.Certificate) (*certmagic.Config, error) {
-			return &magicConf, nil
-		},
-		Logger: a.Logger.Desugar(),
-	})
-	magic := certmagic.New(magicCache, magicConf)
+	a.magicCacheMu.Lock()
+	if a.magicCache == nil {
+		a.magicCache = certmagic.NewCache(certmagic.CacheOptions{
+			GetConfigForCert: func(cert certmagic.Certificate) (*certmagic.Config, error) {
+				return a.setupTLS(dnsservers), nil
+			},
+			Logger: a.Logger.Desugar(),
+		})
+	}
+	certCache := a.magicCache
+	a.magicCacheMu.Unlock()
+	magic := certmagic.New(certCache, magicConf)
 	return magic
 }
-- 
2.39.3 (Apple Git-145)

[maddes-b]

I have the same issue with the current master @27e8251d11ba0a08c9b576fc04d61c1c7ba9b500
What is striking is that it creates 2 caches, but I do not know where these are coming from:

Apr 08 20:13:32 vmanager9064 acme-dns[16782]: 1.7126072127843883e+09        info        maintenance        started background certificate maintenance        {"cache": "0xc000026800"}
Apr 08 20:13:32 vmanager9064 acme-dns[16782]: 1.7126072127844315e+09        info        maintenance        started background certificate maintenance        {"cache": "0xc000026880"}
...
Apr 08 20:43:32 vmanager9064 acme-dns[16782]: 1.712609012784654e+09        error        maintenance        unable to get configuration to manage certificate; unable to renew        {"identifiers": ["<snip>"], "error": "config returned for certificate [<snip>] is not nil and points to different cache; got 0xc000026800, expected 0xc000026880 (this one)"}

This issue is independent of having only a single server and account in api-certs/acme directory, or having multiple servers and/or accounts in there.
Also just updating to latest certmagic v0.20.0 did not help (GOPATH=/tmp/go-acme-dns go get -u github.com/caddyserver/[email protected]). Last bump of certmagic was #334.

On master @6ba9360156b8658dbbd652eea100c11cc098b1f8 I do not see messages for any caches, and do not get any renew errors every 10 minutes.
@joohoi Is this the reason for the other repo at https://github.com/acme-dns/acme-dns/ ? Your personal repo here for development/testing and the other one for production state ?

I found a similar issue for caddyserver/caddy#5162 (with PR caddyserver/caddy#5169 merge caddyserver/caddy@ac96455) also related how to use certmagic.
Do not know if #337 (comment) can be adapted to current master and would fix it.

Update 2024-04-11: I got a solution for the current release, currently testing all cases (renewal, revoked, etc.) plus adding some more debug log messages.

[maddes-b]

Fix for current master developed and tested. Pull rquest is #351

[sndrsmnk]

ty. wonder why this isn't merged yet. manually applied and now i have a new cert again! 👍