Incorrect http_requests_total with multiple Middleware

[rajeshdhanda]

Below is 2 different app along with code and metrics in Grafana.

APP 1

app = FastAPI()
app.middleware("http")(loggingMiddleware)
Instrumentator().instrument(app).expose(app)

Metrics : increase(http_requests_total{job="app-1-api"}[1m])

APP 2

app = FastAPI()
app.middleware("http")(loggingMiddleware)
Instrumentator().instrument(app).expose(app)

@app.middleware("http")
async def auth_middleware(request: Request, call_next):
    global auth_exclude_endpoints
    if request.method != "POST" :
        return await call_next(request)
    ClientAuth = request.headers.get("Authorization")
    auth_resp = await auth_validator(ClientAuth)
    if auth_resp.status_code == 200 :
        return await call_next(request)
    return auth_resp 

Metrics : increase(http_requests_total{job="app-2-api"}[1m])

Upon observation, it's noted that the http_requests_total metric consistently increases in App 2, whereas the traffic distribution to both apps seems similar in reality. Appreciate any insights or suggestions on what might be causing this discrepancy or if there's any missing configuration.

[shrinusivakumar]

@rajeshdhanda I'm experiencing the same issue. It seems that the issue is known. Check out the comment in main.py:

    # Starlette will call app.build_middleware_stack() with every new middleware
    # added, which will call all this again, which will make the registry
    # complain about duplicated metrics.
    #
    # The Python Prometheus client currently doesn't seem to have a way to
    # verify if adding a metric will cause errors or not, so the only way to
    # handle it seems to be with this try block.

If I'm reading this right, then it looks like each middleware will call to add each metric. Adding the first instance of the metrics should be fine. The subsequent calls from each middleware should raise an exception, but I think that exception is not happening in our case. Will continue to dig.

[IceRiverDev]

Hi guys, I also have a question about the http_requests_total. As i understanding, this metric is a counter and the value should always increase, but when i execute query http_requests_total in prometheus dashboard, the graph is like below:

Is the graph normal or not? For my perception the line should always have an increasing trend instead of decreasing.

[Fulmo]

Hi folks, does someone have solution for this? I'm observing the same issue

[akaltsikis]

Hey @trallnag, @Fulmo or anyone above have you been able to find any good workarounds or how to exactly re-produce this locally?

[akaltsikis]

Hey guys @Fulmo @rajeshdhanda @IceRiverDev have you been using gunicorn on your applications ?