Would like updated docs on Unit Testing or a Fakeable/Mockable Polly

[CreepyGnome]

In your doc about Unit Testing with Polly, you provide only broad information, you provide no concrete examples of how.

https://github.com/App-vNext/Polly/wiki/Unit-testing-with-Polly

Could you please provide more specifics and actual examples of how to unit test around Polly and a NoOpPolicy helps in any way.

Either I am using Polly wrong (but I am following your docs on how to use it so I assume I am not) but there is no way to swap out a Policy as you create them via static methods on the Policy class and this is usually inside private areas of classes or at least within a public method. Typically, in my experience at least, Polly is encapsulated within classes so how can I swap out my policy generated based on business rules with this NoOp one?

I don't want to test Polly but I do need to test things around it and make sure Polly is removed. I mean if you had an IPolicy that I can inject via a constructor into a class and then I can inject a NoOp / Null type policy that then pretends to do the work, but that I can then capture the Func that needs to be executed.

This brings up another issue that since the Func contains our code that we would like to test is correct and has not changed there is no mention of how to do that. So this could use concrete examples of how to do this as well.

The doc also talks about using your ISyncPolicy, ISyncPolicy, IAsyncPolicy and IAsyncPolicy interfaces however I see no way to do this in my code as I never am returned any of these interfaces and I don't see anything I use that takes them. Everything is Policy, PolicyBuilder, or a Func for what I use in Polly. So this could use concrete examples as well of how to do what you are briefly describing.

Describe your proposed or preferred solution:

To either update the docs to show how to unit test around Polly or to make Polly be Mockable via interfaces. Currently, you don't have an IPolicy or IPolicyBuilder interfaces if you did and these are what drove the start of all policies then it would be easy to understand and use common unit testing practices to mock them.

Describe any alternative options you've considered:
Only alternatives I can see is passing in a Policy to the constructor of a repository or service and then but expect it to be null always, except for unit tests. If null just use the static methods as they are, otherwise use the one provided however this seems challenging as I still have to call those static methods that are on the abstract base class Policy.

It seems that you expect the developers to mock your concrete classes and didn't really make the Polly library Unit Test friendly so that it can easily be mocked or faked away easily.

So I guess I have been using Polly wrong all these years as this is the first time I've wanted to test a class that uses Polly and I want to mock/fake away Polly as I only test my code, not external code. Which if I am having the concrete examples of how you expect us to write our code in a way that allows Polly to be easily mocked/faked away so that we can focus on testing just our code and not yours.

Any additional info?

No.

[reisenberger]

@CreepyGnome Adding examples to the unit testing page is a very good suggestion (we'll do).

Polly does already provide interfaces permitting the mockability you request. I think the challenge is in discoverability (doco can help address).

Policy configuration syntax does currently declare its return types as concrete classes (example). But every such concrete policy class fulfils interfaces. Taking the same example: here IRetryPolicy<TResult>; and via here indirectly IAsyncPolicy<TResult>. So:

AsyncRetryPolicy<T> : IRetryPolicy<T>, IAsyncPolicy<T>

The step you likely need (we'll document) to get going with Polly and DI for testing, is to declare your polices as those execution interfaces where needed, eg instantiation:

IAsyncPolicy<User> userRepoResiliencePolicy = GetRepoResiliencePolicy<User>();

storing:

IAsyncPolicy<User> UserRepoResiliencePolicy { get; set; }

passing around by DI:

public UserRepository(IAsyncPolicy<User> userRepoResiliencePolicy, ...) { ... }

Every other Polly policy type follows the same pattern. For example NoOpPolicy:

AsyncNoOpPolicy<T> : INoOpPolicy<T>, IAsyncPolicy<T>

This is why you can swap AsyncNoOpPolicy<T> in for AsyncRetryPolicy<T>, if you manage your policies as instances of IAsyncPolicy<T>.

I've used IAsyncPolicy<T>, but the same principles apply with other execution interfaces (ISyncPolicy, ISyncPolicy<T>, IAsyncPolicy, IAsyncPolicy<T>), according to the policy you have configured.

For some immediate examples of that in unit tests, see @bryanjhogan 's excellent blog post. This gives some concise examples of several styles of test, covering angles [1] [2] and [3] of the wiki unit testing page .

I'm hoping this ^ is the magic ingredient you need to get going with Polly and DI for testing, but we will aim to add full examples, and others (DI with PolicyRegistry, and how to play that with testing, merits coverage too).

[CreepyGnome]

@reisenberger I am glad you will be updating the docs, and I already am aware of your existing interfaces as I explained originally. Your examples do not work for my situation, as you are injecting policies into a constructor while, as explained, mine are encapsulated and cannot be injected as they are dynamic and can change based on a number of factors at runtime, including the config file being updated during runtime.

So as I stated I must be doing it wrong even though I am using it like your docs have explained as they don't really show how to do this via constructor injection and still dynamically generate a policy as needed for the situation.

So you claim Polly already provides interfaces permitting the mockability I request. I challenge that this is not the case. As one I didn't request to have static policies that are injected into constructors.

The interfaces I would need are at least an IPolicy and that there are no static methods to generate the policy, this would allow injecting a NoOp type policy that when used to try to generate a policy it would not and the execution would just be a passthrough. IPolicyBuilder may be useful as well in other situations.

Again not static methods as they cannot be easily mocked and extensions methods just cannot be mocked.

My Workaround

The way I worked around this was that since the configuration that is injected, I have the retry count, the jitter factors all default to 0 which makes no retries and then there are no jitter even if there was.

This then lets the Func that will be passed to the ExecuteAsync will run always and only once. I can then test around Polly just testing the mocks that are eventually used in the ExecuteAsync Func can be tested for methods calls and data passed, etc.

This was clearly obvious at first, but I don't' want to test Polly so I trust its ExecuteAsync will always work as expected so I can then ignore it and by nullifying how the policy works based on zero'd out config values it is just a pass-through essentially.

So again, maybe I am doing it wrong by using encapsulation and even having dynamic policies that can change based on need and configuration and therefore we don't' cache the policy and new one is generated every time one is needed.

If there is an issue with having a lot of temporary policy objects for brief periods of time let me know. I am assuming Polly cleans up after itself as well and that these are not becoming a memory leak/burden. If so let me know I will reconsider the design and maybe cache until they change and then only generate when needed, but I like to keep it simple and only add complexity when needed.

[reisenberger]

@CreepyGnome What is/are the specific goal/s of the tests where challenges arise? Is it around the dynamic generation of policies within the class?

Have I understood correctly that this class (system-under-test) is a repository (?- or?) and that it manufactures the policy (from config? passed in params? other factors, biz rules, also affect the dynamic gen?). And class then/also executes through the policy on some underlying data store? (correct me if this is wrong- I'm sketching, to form a picture). Please feel free to post the actual code, or representative edit (it would be particularly helpful to get an idea of the signature of the ctor and method under test). Thanks.

[reisenberger]

Re temporary policy objects:

• no issue with stateless policies except the usual allocation/GC (policies are fairly light)
• circuit-breaker and bulkhead are stateful - with these you need to use the same policy instance across calls-to-be-governed-in-common, for the policy to fulfil its function. Concretely: circuit-breaker counts failures, so the same instance needs to be used across the multiple calls whose failures/successes drive circuit state. Bulkhead limits parallelism - the same instance needs to be used across the calls whose parallelism is to be limited.

[CreepyGnome]

@reisenberger I guess the goal was or what I expected was to be able to inject an Interface for what actually creates a policy not an interface for a policy.

Thanks for the details on stateless policies I feel a little better know they are fairly light and if we do get an issue I can implement a caching pattern.

Yes the SUT is a repository to a mongoDB collection, and I use the policy to handle the throttling required for hitting our limits. As well as general retry for specific errors. The policies are currently all stateless. I just want test the repo is calling and doing the right things and removing all external dependies, aka MongoDB, Policies, etc. as they don't matter for the tests.

Yes, there is a json appconfig with a section that can be changed at runtime. There are rare but potential variables that can cause a policy to be different occasionally.

So using composition the repository contains a MongoDbRetrier which manages the business rules, for generating policies including managing the backoff with jitter for MongoDb repositories. This has a couple "Retry" methods that create the policy based on the current rules and then executes it. The repository then wraps every call to our collection with one of the retriers retry methods.

an example ctor for the repository is

public SomeRepository(IOptionsSnapshot<SomeRepoSettings> settings, ILogger<SomeRepository> logger, IMongoClient client) 

The constructor creates a new MongoDbRetrier based on the settings that it ensures follows the rules during a retry. The repository is not a singleton so it created every call to a Controller and it pass to a service that passes to the repository. So the snapshot will ensure that the latest config settings are used each time.

I asked and it takes time for an approval to post our code in public (I know crazy) and didn't have time to remake an "on my time" representative sample. So I tried to explain it well I hope it helps. If not I can find time to create a representative sample that is good enough.

[reisenberger]

I asked and it takes time for an approval to post our code in public (I know crazy) and didn't have time to remake an "on my time" representative sample. So I tried to explain it well I hope it helps.

Thanks @CreepyGnome , the description just posted gives a great picture (no need to push on getting approvals for posting actual code if problematic - not unusual that companies have may have a view on that).

I'll post back with some concrete suggestions around the testing soon.

[reisenberger]

Thanks @CreepyGnome for all the extra info - it's a great use case for hearing how people are pushing the envelope with Polly.

Yes: constructed Polly policies fulfil interfaces, but the policy builder syntax is static or concrete-builder-class methods, and so not amenable to mocking as a strategy for testing.

I'll post thoughts separately around options for converting Polly's builder syntax to an interface/instance-method approach (longer topic), but TL;DR it wouldn't happen overnight; so this concentrates on immediate practical suggestions for testing the dynamic policy generation code you have.

---

Generating policies dynamically at runtime need not be incompatible with a DI-based approach (notwithstanding aspects to your code I may be unaware). You could consider pulling a policy factory out of the code described, something like:

interface IPolicyFactory {
    IAsyncPolicy CreatePolicy();
}

(I'm not sure from the code described whether your repositories are non-generic SomeRepository or generic SomeRepository<T> or indeed, either way, whether the policies used are non-generic IAsyncPolicy or generic IAsyncPolicy<T>. I'll go with non-generic; adapt to IAsyncPolicy<T> if needed.)

The ctor for SomeRepository would then become something like:

public SomeRepository(IPolicyFactory policyFactory, ILogger<SomeRepository> logger, IMongoClient client) 

with some concrete implementation RepositoryPolicyFactory : IPolicyFactory taking the IOptionsSnapshot<SomeRepoSettings> settings in its ctor:

public RepositoryPolicyFactory(IOptionsSnapshot<SomeRepoSettings> settings, /* any any other params needed */)

ie the IPolicyFactory becomes responsible for/encapsulates manufacturing policies, while the SomeRepository (or MongoDbRetrier, depending on your exact code) is responsible for executing through the policy.

You could then isolate Polly out of repository tests where needed, by injecting a NoOpPolicyFactory into SomeRepository for those tests, where:

public class NoOpPolicyFactory : IPolicyFactory
{
    IAsyncPolicy CreatePolicy() => Policy.NoOpAsync();
}

You could also construct a MockPolicyFactory:

public class MockPolicyFactory : IPolicyFactory
{
    private Mock<IAsyncPolicy> mockPolicy;
    public MockPolicyfactory(Mock<IAsyncPolicy> mockPolicy) => this.mockPolicy = mockPolicy;
    IAsyncPolicy CreatePolicy() => mockPolicy.Object;
}

A test can then pass a MockPolicyFactory instance into SomeRepository and assert that SomeRepository (or its internal MongoRetrier) does execute through the policy the IPolicyFactory supplies.

This is essentially suggestion 1 from our unit-testing wiki page but applied with policy factories - since you generate policies dynamically.

Hope this is useful - the aim is to show concretely you can still take a DI approach to isolating concerns when generating policies dynamically.

[reisenberger]

The preceding suggestions still leave the problem of how to test the dynamic policy generation (ie your real implementation RepositoryPolicyFactory : IPolicyFactory or similar), absent mockability.

I'm assuming the emphasis on mockability was in order to test how policies were generated or used, using mock.Verify(...) or similar (given the other main use cases for mocking - stubbing out expensive objects or downstream systems - don't apply to policies). Say if I've inferred wrongly.

Absent mockability of PolicyBuilder, we have to test what has been generated, not how it is generated.

That a retry policy handles a given exception N number of times could be tested with a helper method like this:

public static bool HandlesN(this RetryPolicy retryPolicy, int n, Exception ex)
{
    var catch = Policy<bool>.Handle<Exception>().Fallback(false);
    int i = 0;
    return catch.Wrap(retryPolicy).Execute(() => i++ < n ? throw ex : true);
}

(EDIT: forgot the case in question may in async; if you need an async example, let me know)

So if the responsibility of your RepositoryPolicyFactory : IPolicyFactory is to generate retry policies from config, which will handle given exceptions given numbers of times, that could be used to test it does.

Assert.IsTrue(myRetryPolicy.HandlesN(3, new MongoClientException());

Based on earlier comments and your question here, I'll address the case that you have, or may have in future, code to dynamically create a PolicyBuilder to handle exceptions of a given Type instance, like requested here.

That a given concrete PolicyBuilder instance handles a given exception could be tested with a helper method like this:

public static bool Handles(this PolicyBuilder builder, Exception ex)
{
    var catchEverythingElse = Policy<bool>.Handle<Exception>().Fallback(false);
    var builderTester = builder.Retry(1);
    int i = 0;
    return catchEverythingElse.Wrap(builderTester).Execute(() => i++ == 0 ? throw ex : true);
}
// or just
public static bool Handles(this PolicyBuilder builder, Exception ex) => builder.Retry(1).HandlesN(1, ex);

Using this code just because it is a concrete example to refer to (you could easily make that code instance methods if you prefer), eg:

var handleClause = PolicyExtensions.Handle(typeof(MongoClientException));
Assert.IsTrue(handleClause.Handles(new MongoClientException());
Assert.IsFalse(handleClause.Handles(new SqlException());

If you were to take the ^^ previously-suggested approach of factoring out an IPolicyFactory, you could split it further, pulling an IPolicyBuilderFactory out of it for the parts that build the handle-clauses of the policies:

interface IPolicyBuilderFactory {
    PolicyBuilder CreatePolicyBuilder();
}

The concrete implementations of IPolicyBuilderFactory could then be tested with the public static bool Handles(this PolicyBuilder builder, Exception ex) extension method.

Hope useful - aim is to demonstrate some concrete helpers for testing the dynamic policy generation discussed.

[reisenberger]

To whether Polly could adopt an instance/interface-based configuration approach, or introduce IPolicyBuilder:

One observation is that adopting IPolicyBuilder would mean that IPolicyBuilder would have to expose some properties which are currently internal - ExceptionPredicates and ResultPredicates.

Or alternatively keep them encapsulated but expose only methods that policies could use to evaluate exceptions/results against them, eg:

bool IsExceptionHandled(Exception ex)

A problem is that this exposes methods on IPolicyBuilder that are not actually part of the user-experience we want of the builder syntax. Users could be led down blind-alleys/dead-ends, as they try to construct the policy:

Policy
    .Handle<FooException>()
    .ExceptionPredicates
    . // what do I do now?

or

Policy
    .Handle<FooException>()
    .IsExceptionHandled(/* some exception instance */) // very confusing! - is this part of policy-building syntax or not?
    . // what do I do now?

This may not be insuperable, but is a concern. An advantage of the current, 'closed' concrete builder classes is that there are no blind alleys in the syntax - users are inexorably guided to construct a finished policy.

(EditorBrowsableAttribute could be an angle on this - but it is not consistently honored.)

(Explicit interface implementation to hide members could be another angle - but explicit interface implementation only works on classes/when the type in question is returned as a concrete class - and the goal of any change is specifically to work with an interface IPolicyBuilder instead.)

It's also possible that a .Net-core-style indented configuration syntax could avoid/minimise some of that problem:

Policy.Handle(Action<IHandleConfiguration> configure)

leading to usage like:

Policy.Handle(h => h.Handle<FooException>())

IHandleConfiguration is separate from IPolicyBuilder, so the blind-alleys do not arise. This is the best candidate I have so far for such a transition, but it's less snappy, and is a major breaking change.

EDIT: Moving fully to an instance/interface-based approach could read:

new PolicyBuilder()
    .Handle<Fooexception>()
    // etc

(This would be a major breaking change.)

---

This is not a definitive statement on whether we go to IPolicyBuilder, just some initial observations.

Definitely interested in community suggestions too - if there is some obvious better syntax we are missing?

[reisenberger]

Hi @CreepyGnome, Is there anything else we can follow up on, after the unit testing suggestions above?

Good shout on improving the unit-testing documentation: that's been done, in this page, with example code.

Recasting the Policy .Handle<>() clauses to instance/inteface-based approaches: I've lodged this in the roadmap with further commentary, as well as pointers back to here.

[reisenberger]

Closing due to no further follow-up at this time. However, if you come back to this at a later date, just post again and we can pick up the conversation, happy to help further.