InvalidOperationException being thrown from Unleash.Scheduling.SystemTimerScheduledTaskManager.ConfigureTask

[dcook-net]

Describe the bug
We are seeing the error "Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct." being thrown from Unleash.Scheduling.SystemTimerScheduledTaskManager.ConfigureTask when it calls Dictionary.FindValue.

From looking at the code, I assume this is becuase of the way the SystemTimerScheduledTaskManager mutates the Timers Dictionary.

To Reproduce

We are using v4.1.7
Polling is set to 120 seconds

We are calling client.IsEnabled within a loop for 4 flag names, like so:

client?.IsEnabled(toggleName, unleashContext, defaultValue) ?? defaultValue;

The client is a singleton, as described in the documentation

I'm happy to raise a PR chaning Timers the Dictionary to a ConcurrentDictionary if this is an acceptable solution.

[dcook-net]

Upon closer inspection, it appears that despite the app using an unleash client as a singleton, we have a HealthCheck service that periodially creates a seperate instance, and determines the health of the connection by whether that client was successfully creatd or not (ie, is it null or not)?

I believe all the instances of this error originate from this health check.

For info, there are very few of these. In 10 period, we've only seen 6 instances of this.

[daveleek]

Thank you for getting back to us on what seems to be the root cause for the issues you've experienced! I'll close this issue for now, but you're welcome to reopen or open a new issue if this didn't address the issue or you experience any new issues!