Message queue stuck with processing (hangs)

[XBeg9]

Hi, we are seeing issues with netherite backend when messages are stuck with processing and orchestrator scheduled, but never does the job and the whole process hangs. Any ideas what this could be?

[nls-jbarntish]

Is it intermittent or does it happen each time you try to execute? And is it happening using the default (blob storage) provider?

Just as a shot in the dark here: at one point in a fan-out, I accidentally used await Task.WaitAll() instead of await Task.WhenAll(). This caused the orchestrator to execute fine up until this fan out and it just hung there doing nothing. No logging, no status updates, no activity executions, etc. Took me a while to figure that one out, as it's easy to overlook.

Another thing to check: if you have a lot of Event Hub throttling going on, try upping the TUs. This smoothed out our orchestrations quite a bit.

[XBeg9]

it's intermittent for large activity batches (100 per sub-orchestrator). I have auto inflate on the event hub, but we are not using premium plan.

[jbarntish]

1.4.0 was just released today, might try upgrading to that and re-testing. I am seeing significant improvements in event hub throughput and there are some underlying memory fixes as well.

You'll need to fully reset your existing task hub after upgrading. I tried upgrading without reset and the orchestrations failed almost immediately.