Wide range of throughput numbers using my large test case #240
Comments
|
There can be many reasons why throughput fluctuates. If you give me the name and region of your function app, and a start/end time for the experiment(s), I can a look at our internal telemetry to see if I can determine a cause. If I were to just guess, my guess would be that it has to do with the load balancing of activities between nodes. This may or may not be the actual reason. One thing you can try is to turn off the load balancing altogether, using the setting |
|
Thanks I'll try the "Local" setting later and report any differences. |
|
Sorry, I decided I should keep this one open until after I've had a chance to test with the 1.4.0 version. |
The new infamous test case I'm using for Netherite that launches now 200 orchestrators and 200 activities each, seems to report widely varying throughput numbers, even after scale up.
I'm wondering if something is working differently than expected.
So over the course of my looping sample, if I start from idle, I'd expect to see slower throughput, then faster and faster, then some sort of stable or close-to-stable number of items processed/second.
What I'm seeing varies quite widely, from 14000/second then sometimes down to 2000/second and less.
See below... This is a test of 40000 items (200/200) in a loop, so the invocations are right after each other.
My next version will be so good as to log the wall time, sorry about that.
Included are a few other numbers to perhaps help the analysis.
Min and max orchestrator Dequeue delay are the times that it takes between the orchestrator being initiated from StartSubOrchestrator and the time the orchestrator code starts to run.
Similarly with Min and Max activity dequeue delay.
Out dequeue delay is the time from the return statement in the orchestrator or activity to the time the parent code gets control.
Is it expected that the times should vary so widely? I'm using a copy of the benchmark host.json in my tests.
{ "version": "2.0", "logging": { "logLevel": { // ------ NOTE: // The log levels in this section // - determine what is displayed in the console when running func.exe locally // - determine what is stored and billed by Application Insights // But they do not fully control the logging: // - have no effect on what is collected by internal analytics // - have no effect on what is traced to the console when running in a container // - have no effect on what is shown in the Live Metrics side panel in the portal // ---- Per-invocation framework-generated logging //"Host.Triggers.DurableTask": "Information", // use this setting if you need analytics in the portal "Host.Triggers.DurableTask": "Warning", // use this setting otherwise // ---- Per-invocation application-generated logging //"Function": "Information", // use this setting for small-scale debugging only "Function": "Warning", // use this setting when running perf tests // --- the levels below are used to control the Netherite tracing. "DurableTask.Netherite": "Information", "DurableTask.Netherite.FasterStorage": "Warning", "DurableTask.Netherite.FasterStorage.Performance": "Error", "DurableTask.Netherite.EventHubsTransport": "Warning", "DurableTask.Netherite.Events": "Warning", "DurableTask.Netherite.WorkItems": "Warning", "DurableTask.Netherite.Client": "Warning", "DurableTask.Netherite.LoadMonitor": "Warning" }, "applicationInsights": { "sampling": { "isEnabled": false } } }, "extensions": { "durableTask": { // "hubName": "perftests", "UseGracefulShutdown": true, // helps to avoid lengthy waits for lease expiration "maxConcurrentActivityFunctions": "100", "maxConcurrentOrchestratorFunctions": "100", "storageProvider": { "type": "Netherite", // "type": "AzureStorage", "StorageConnectionName": "AzureWebJobsStorage", "EventHubsConnectionName": "EventHubsConnection", "TakeStateCheckpointWhenStoppingPartition": "true", "MaxNumberBytesBetweenCheckpoints": "20000000", "MaxNumberEventsBetweenCheckpoints": "10000", "IdleCheckpointFrequencyMs": "60000", "PackPartitionTaskMessages": 100, "PartitionCount": 12, // set this to false to force replay on each orchestration step "CacheOrchestrationCursors": true, // set this to control the max size of the orchestration instance cache // "InstanceCacheSizeMB": "50", // set this to true to use the PSF support in Faster. Will soon be obsolete. "UsePSFQueries": "false", // set this to true to bypass Faster and instead use Azure blobs for the object store. Not for production use. "UseAlternateObjectStore": "false", // set this to "true" to disable all pipelining "PersistStepsFirst": false, // set this to x to inject faults with probability 1/x //"FaultInjectionProbability": 2000, // set this to true to attach replay checker //"AttachReplayChecker": true, // set this to true to attach replay checker //"AttachCacheDebugger": true, // can change this to use a different table, or blobs //"LoadInformationAzureTableName": "", // set this to "Scripted" to control the scenario with a partition script // or to "ClientOnly" to run only the client "PartitionManagement": "EventProcessorHost", // set this to "Local" to disable the global activity distribution algorithm // options: "Local", "Static", "Locavore" "ActivityScheduler": "Locavore", // controls whether the instance ids are kept in memory. If false, paged queries are not supported. "KeepInstanceIdsInMemory": true, // The log level limits below control the production of log events by the various components. // it limits production, not just consumption, of the events, so it can be used to prevent overheads. // "Debug" is a reasonable setting, as it allows troubleshooting without impacting perf too much. "LogLevelLimit": "Debug", "StorageLogLevelLimit": "Debug", "TransportLogLevelLimit": "Debug", "EventLogLevelLimit": "Debug", "WorkItemLogLevelLimit": "Debug", "ClientLogLevelLimit": "Debug", "LoadMonitorLogLevelLimit": "Debug", // the following can be used to collectd and direct trace output to additional specific sinks // which is useful in a testing and debugging context "TraceToConsole": false, "TraceToBlob": false } } } }The text was updated successfully, but these errors were encountered: