Document testing strategies and the use of the GuiTestAssistant

[mdickinson]

We need to document better how to test code that uses Traits Futures.

• General testing strategies for code that's using Traits Futures, and possible pitfalls
• Use of PyFace's GuiTestAssistant
• Use of the GuiTestAssistant that's in Traits Futures (should we make this public, and perhaps rename it?)

[mdickinson]

An example motivating problem (taken from a real project that was using Traits Futures): you're writing a UI that fires off various background tasks and gets some CallFuture objects back. You want to write unit tests for the way that the UI reacts to the state / result / exception changes on the future, but to keep the unit tests fast and simple, you don't want to fire off real background tasks.

[mdickinson]

The main point that needs documenting is the need to run the event loop until the desired outcome occurs.

[stpotter16]

@mdickinson FWIW, we stumbled through writing tests for our application's usage of Traits Futures. Relaying that here hoping it will help with deciding the priority of this issue. Thanks!

[mdickinson]

@stpotter16 Please could you take a look at #278 and see (a) if it helps, and (b) what additions / modifications would be useful for your use-case?

[mdickinson]

This issue was partially addressed in #278. I'm leaving the issue open while we consider whether there's more than should be added to that documentation.

For "Use of the GuiTestAssistant that's in Traits Futures", that should likely be a separate issue. The example in #278 used Pyface's GuiTestAssistant.

[mdickinson]

@stpotter16 Actually, on second thoughts, rather than reviewing the already-merged PR, it would be better to go straight to the rendered docs: https://docs.enthought.com/traits-futures/dev/guide/testing.html

[stpotter16]

@mdickinson Docs are useful as is. I've created an issue on our application's repo to update the affected tests and will link the docs there. Thanks!

[kitchoi]

On testing with traits-futures, just an idea, would it be possible to use an implementation of concurrent.futures.Executor where one can manually drive the execution on a per-task level, and then the tests can be set up to run with this executor?

e.g. concurrent.futures.Executor.set_result (and friends) are exposed to be used for testing and executor implementations.

The difficulty I ran into with GuiTestAssistant is that since it uses a poll timer, if my test should have two asynchronous tasks, I have not been able to use GuiTestAssistant to test my application state after the first task is completed and before the second task is completed: It is all a bit of lucky draw.

(And other minor details is that GuiTestAssistant makes the test slower because of the polling. It is only a bit, but it adds up when you are zealous about testing.)

The semi-private methods mentioned in #225 are useful to drive changes on a per-future level. However it requires one to dive into, and sometimes reimplementing, the call that where submitted to be run asynchronously. They also require a more detailed understanding of the lifecycle of states in traits-futures: All-in-all, more implementation detail-level knowledge is required.

[mdickinson]

if my test should have two asynchronous tasks, I have not been able to use GuiTestAssistant to test my application state after the first task is completed and before the second task is completed: It is all a bit of lucky draw

Do you have control over the tasks being tested? A solution I often use for this situation is to use threading.Event objects to coordinate the timings between the tasks. (E.g., in the second task, insert a signal.wait(), and then do a signal.set() from the main thread once the first task has completed and you've applied the required tests to its state.)

[mdickinson]

And other minor details is that GuiTestAssistant makes the test slower because of the polling.

Yes, that's one of the reasons that Traits Futures uses its own GuiTestAssistant that doesn't poll. Ideally, we'd make that public (and also rename it, to avoid confusion). That's on the roadmap for the next release.

[mdickinson]

That's on the roadmap for the next release.

Whoops, sorry. It wasn't, but it is now. I thought we had a separate issue for that, but there's only this issue.

[mdickinson]

Opened #304 for the specific task of exposing GuiTestAssistant. See also #252.

[mdickinson]

would it be possible to use an implementation of concurrent.futures.Executor where one can manually drive the execution on a per-task level

It's possible, and we've done things like this in the past, but it's not trivial. encore has a SynchronousExecutor: https://github.com/enthought/encore/blob/master/encore/concurrent/futures/synchronous.py. It may be worth seeing if it's possible to provide a context that uses that. (Though it's a bit awkward: I don't really want to grow a hard dependency on encore in this package.)

[kitchoi]

I see why the SynchronousExecutor would be awkward to be used in traits-futures context: When SynchronousExecutor.submit returns, the task is already finished (done, in traits-futures terminology). It would be harder to test application when the task (1) has not started (2) is waiting (3) ... all the different states. Just a brainstorm to throw some ideas (motivated by existing test patterns), perhaps the executor can just keep track of the work item when they were submitted so that one can do something like this?:

# in the application code 
future = submit_call(executor, my_very_slow_function)

# in test setup, use the test executor for the application
test_executor = SomeImplementationOfExecutorForTest()
app = MyApp(executor=test_executor)

# in test code, if one can get hold of the future instance (which is likely the case)
test_executor.resolve(future)  # this sets the result/exception

[mdickinson]

Deferring this.