How to add timeouts for each retry attempt? Will current code support that?

[pulkitent]

Hi team,

Apologies if this has been asked already but I was unable to find the answer.

I am using Polly library to implement a retrial mechanism for a REST API call with timeouts using below code. I would like to know the whether the timeout I have added in the ResiliencePipeline using AddTimeout() is applicable at each retrial level or is it applicable for last retrial only?

RetryStrategyOptions retryStrategyOptions = new RetryStrategyOptions
{
    ShouldHandle = new PredicateBuilder().Handle<Exception>(IsTransient),
    MaxRetryAttempts = 3,
    DelayGenerator = static args =>
    {
        var delay = args.AttemptNumber switch
        {
            0 => TimeSpan.Zero,
            1 => TimeSpan.FromSeconds(60),
            2 => TimeSpan.FromSeconds(120),
            _ => TimeSpan.FromSeconds(180)
        };
        return new ValueTask<TimeSpan?>(delay);
    }
};

ResiliencePipeline pipeline = new ResiliencePipelineBuilder()
    .AddRetry(retryStrategyOptions)
    .AddTimeout(TimeSpan.FromSeconds(10)) // Some Value
    .Build();

pipeline.ExecuteAsync(async (cancellationToken) => { //Some REST API call } );

For example:

Case 1
Attempt 0: Occured at 2:00:00 AM but we didn't receive any response then will there be a timeout at 2:00:10 AM ?
Attempt 1: Occured at 2:01:00 AM but we didn't receive any response then will there be a timeout at 2:01:10 AM ?
Attempt 2: Occured at 2:02:00 AM but we didn't receive any response then will there be a timeout at 2:02:10 AM ?

OR

Case 2
Attempt 0: Occured at 2:00:00 AM but we didn't receive any response
Attempt 1: Occured at 2:01:00 AM but we didn't receive any response
Attempt 2: Occured at 2:02:00 AM but we didn't receive any response then will there be a timeout at 2:02:10 AM ?

OR

There is something else?

[peter-csala]

The registration order of the strategies does matter.

In other words you will have very different resilience behaviour if you register your retry and timeout strategies like this

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(....)
    .AddTimeout(...)
    .Build();

or like that

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(...)
    .AddRetry(....)
    .Build();

Here we have documented these scenarios: https://www.pollydocs.org/pipelines/index.html#diagrams

I hope the documentation clarifies how you should register them. If not, please let me know.

[peter-csala]

Your setup (Retry the outer, Timeout the inner) means the following: For each retry attempt you have a separate timeout. That means you don't have an overall timeout for all the retry attempts.

[martincostello]

With Timeout first, the timeout will apply to everything that happens subsequently, so in your example everything would timeout at 10 seconds. This would effectively mean only 1 retry, assuming the first call failed within 10 seconds, as the 60 second delay is greater than 10 seconds, so it would never happen as the timeout would occur before the next attempt.

With Timeout last, the timeout would apply to each call made by the (re)try.

[pulkitent]

Thanks @peter-csala & @martincostello, and if in below code my timeout is of 5 minutes then

Attempt 0 : Assumed it occured at 2:00:00 AM & API call failed.
Attempt 1 (Retrial) : Will occur at 2:01:00 AM & let's say API call failed again.
Attempt 2 (Retrial) : Will occured at 2:02:00 AM & API call failed again.

Now at 2:05:00 AM (ideally) this operation will timeout? Is my understanding correct?

RetryStrategyOptions retryStrategyOptions = new RetryStrategyOptions
{
    ShouldHandle = new PredicateBuilder().Handle<Exception>(IsTransient),
    MaxRetryAttempts = 3,
    DelayGenerator = static args =>
    {
        var delay = args.AttemptNumber switch
        {
            0 => TimeSpan.Zero,
            1 => TimeSpan.FromSeconds(60),
            2 => TimeSpan.FromSeconds(120),
            _ => TimeSpan.FromSeconds(180)
        };
        return new ValueTask<TimeSpan?>(delay);
    }
};

ResiliencePipeline pipeline = new ResiliencePipelineBuilder()
    .AddRetry(retryStrategyOptions)
    .AddTimeout(TimeSpan.FromSeconds(300)) // 5 minutes
    .Build();

pipeline.ExecuteAsync(async (cancellationToken) => { // Some REST API call } );

[peter-csala]

Now at 2:05:00 AM (ideally) this operation will timeout? Is my understanding correct?

No. Since you have called the AddTimeout after the AddRetry that's why the timeout strategy is applied per retry attempt. So, in this particular case each retry attempt has a five minutes window to complete.

In worst case scenario this means the following:

1. 0th attempt (original request) starts at 2:00 and ends at 2:05 because it times out
2. The retry strategy sleeps 0 minute
3. 1st attempt (first retry attempt) starts at 2:05 and ends at 2:10 because it times out
4. The retry strategy sleeps 1 minute
5. 2nd attempt (second retry attempt) starts at 2:11 and ends at 2:16 because it times out
6. The retry strategy sleeps 2 minutes
7. 2nd attempt (second retry attempt) starts at 2:18 and ends at 2:23 because it times out
8. The retry strategy runs out of retry attempts

---

If you want to have 1 minute timeout per retry attempt and 5 minutes overall timeout then you should have something like this:

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromMinutes(5)) // overaching timeout
    .AddRetry(...)
    .AddTimeout(TimeSpan.FromMinutes(1)) // per retry attempt timeout
    .Build();

[pulkitent]

Thanks @peter-csala, for a such a wonderful explaination. If I want to unit test the scenario where all the attempts are exhausted because of timeouts.

Any code references you have?

If I mock my API call using MOQ & add Thread.Sleep() in that will that help me to repplicate?

[peter-csala]

You don't need to use Thread.Sleep you can use the FakeTimeProvider to mimic time advancement.

using System;
using System.Threading;
using System.Threading.Tasks;

using Microsoft.Extensions.Time.Testing;
using Polly;
					
public class Program
{
	public static async Task Main()
	{
		var timeProvider = new FakeTimeProvider();
		var pipeline = new ResiliencePipelineBuilder { TimeProvider = timeProvider }
			.AddTimeout(TimeSpan.FromMilliseconds(100))
			.Build();
		
		await pipeline.ExecuteAsync(async ct =>
		{
			timeProvider.Advance(TimeSpan.FromSeconds(20));
			await Task.Delay(TimeSpan.FromSeconds(10), timeProvider, ct);

		}, CancellationToken.None);
	}
}

This will result in a TimeoutRejectedException.

Dotnet fiddle: https://dotnetfiddle.net/NvIRdd