FluentBit - Sending queue is full when trying to send logs.

[simlarsen]

����sending queue is full
[2025/01/08 20:13:48] [ warn] [engine] failed to flush chunk '1-1736367227.589268651.flb', retry in 11 seconds: task_id=10, input=tail.0 > output=opentelemetry.0 (out_id=0)
[2025/01/08 20:13:49] [ info] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=200

[edgarrmondragon]

fwiw the error does seem to be coming from the OTEL collector used by OneUptime: https://github.com/open-telemetry/opentelemetry-collector/blob/12b0369f16c0f7c5e45832dc05bfc2960975cd50/exporter/exporterqueue/queue.go#L18

[simlarsen]

@edgarrmondragon Can you please look into this. This should now be fixed. Please feel free to reopen this issue if you think this is still not fixed.

[edgarrmondragon]

Hey @simlarsen, I think we're still experiencing this:

[2025/01/14 12:56:27] [error] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=503
����sending queue is full
[2025/01/14 12:56:27] [ warn] [engine] failed to flush chunk '1-1736859386.398660857.flb', retry in 7 seconds: task_id=1, input=tail.0 > output=opentelemetry.0 (out_id=0)
[2025/01/14 12:56:28] [ info] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=200

[simlarsen]

@edgarrmondragon Looking into this.

[simlarsen]

Can you please let me know if this is fixed for you now? If not let me know and I can re-open this issue.

[edgarrmondragon]

Thank you @simlarsen, we're currently monitoring and I'll let you know if the error no longer rears its heads in the coming days.

(So far things are looking good 🎉)

[edgarrmondragon]

@simlarsen we're still occasionally seeing the 503s and sending queue is full errors:

[2025/01/17 16:21:06] [error] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=503
��sending queue is full
[2025/01/17 16:21:06] [ warn] [engine] failed to flush chunk '1-1737130865.520098411.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=opentelemetry.0 (out_id=0)
[2025/01/17 16:21:07] [error] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=503
��sending queue is full
[2025/01/17 16:21:07] [ warn] [engine] failed to flush chunk '1-1737130866.527146668.flb', retry in 8 seconds: task_id=1, input=tail.0 > output=opentelemetry.0 (out_id=0)
[2025/01/17 16:21:08] [ info] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=200


[2025/01/17 16:21:09] [error] [output:opentelemetry:opentelemetry.0] oneuptime.com:443, HTTP status=503
��sending queue is full
[2025/01/17 16:21:09] [ warn] [engine] failed to flush chunk '1-1737130868.518212564.flb', retry in 9 seconds: task_id=2, input=tail.0 > output=opentelemetry.0 (out_id=0

[simlarsen]

@edgarrmondragon got it! Let me look into this and investigate.

[simlarsen]

@edgarrmondragon workign on this today / tomorrow. Will have an update soon.