Issues With the System Testing Framework

We make use of a framework taken from PIPY for our system testing. Generally speaking it is a good idea to use external tools as it reduces maintenance requirements. However as we gain experience through use it is becoming apparent that the tool we have chosen isn't perfectly matched with our requirements.

We will continue to use it for the moment but this page is here to collect issues so we can make a decission in the future. There are three obvious possible decissions to be made. Either improve the current frame work, Find a replacement or implement our own.

Issues

1. Console output is very verbose meaning that a failing test early in the sequence leads to a lot of scrolling to determine cause.
2. Console output doesn't seem to take into account tests running in parallel leaving the output from both tests interleaved. This makes comprehension harder.
3. Internally the framework highjacks docstrings for console output. This leads the source to be odd as some things which should be in the doc string but shouldn't be in the ouptut have to be included as comments instead.