Too many Redis calls happen randomly

[SlavaSkvortsov]

From time to time we notice that ARQ starts to produce a lot of calls to Redis. It consumes a lot of connections and puts a lot of load on Redis
Here how it looks like during the incident:

And most of the time it looks like this:

It feels like it starts to attempt to execute jobs over and over nonstop. I think this behaviour could happen when you have queue_read_limit lower than max jobs on all servers, but that's not the case for us.

We're using:

python==3.11.3

arq==0.26.0
redis==4.6.0

Unfortunately, I can't reproduce the issue locally, but it happens a couple times a month on prod