Performance inconsistencies with orchestrations

[dylanhouston7]

Hi,

I have been working with durable functions for a while. Much thanks to the dev team for fixing the issue I raised earlier :) #2162

I'm noticing that when running orchestrations in a deployed Azure environment, I am noticing very different run durations for the same set of activity functions/orchestrations. I am using an EP1 app service plan. There is not a lot of I/O in these orchestrations.

In my brief analysis, it appears that it is taking a while for the activity functions to pick up the queued messages to process. The execution times of the functions themselves do not appear to be problematic.

My host.json file is as follows:

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "%TaskHub%",
      "extendedSessionsEnabled": true,
      "extendedSessionIdleTimeoutInSeconds": 300,
      "useGracefulShutdown": true
    }
  },
  "functionTimeout": "00:00:30",
  "retry": {
    "strategy": "fixedDelay",
    "maxRetryCount": 3,
    "delayInterval": "00:00:30"
  },
  "logging": {
    "logLevel": {
      "default": "Information",
      "Microsoft": "Warning",
      "Azure.Messaging.ServiceBus": "Information",
      "Microsoft.Azure.WebJobs.Extensions.DurableTask": "Warning",
      "DurableTask.AzureStorage": "Warning",
    },
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  }
}

Example orchestrations:

This one took less than 3 seconds to complete, orchestration id 1d0d61b5a83249e2bdf0ea0921a7b5f6

190aa178201b4b049967115e035b9ca2 is another example of one that took less than 3 seconds to complete.

This one took a little less than 6 minutes to complete, a big difference from 3 seconds.
32258d8061824ed6b0895e05f302d5d5

Here are another couple recent orchestrations from a couple days ago that took much longer than expected. One even took ~11 minutes, and the other took 7.
7245c7944cd6410ca9145955d886cfd9
a94432ec02bf4edea62127866aedf77c

Could I get some insight as to why it is taking so long for some of my orchestrations to complete, while others are going through in a matter of seconds? I saw earlier on GitHub #606
that a delay of 30 seconds is sometimes to be expected, but ~6-12 minute orchestrations is not very good.

Thanks,
Dylan

[davidmrdavid]

Thanks for reaching out @dylanhouston7. I'm sharing this internally for review. We'll get back to you on it as soon as possible. It is indeed strange to see such variance in execution time.

[dylanhouston7]

Thanks for reaching out @dylanhouston7. I'm sharing this internally for review. We'll get back to you on it as soon as possible. It is indeed strange to see such variance in execution time.

Thanks for taking a look @davidmrdavid . Here a couple new orchestrations just ran today along with what I see in the Storage Explorer for both of them.

fe7afc46acd84746b9debd2d8d486bcd - ~ 8 minutes

044df39b7ca7414bae5765820ad06ea2 - ~7 minutes

[dylanhouston7]

Hi @davidmrdavid , just thought I'd post two more. One that took about a minute, and one that took about 7 minutes. Both were run within the past two days.

~ 20 seconds - 1c381b2f3eeb40de8570becd7be02275

Almost 7 minutes - 044df39b7ca7414bae5765820ad06ea2