Unit Test Coverage

Due to issues surrounding resourcing 3rd party tools we do not currently integrate unit test coverage into our pull request actions. This does not stop developers using some tools manually though.

Generating the Report

Simply passing the appropriate argument will cause PyTest to generate a coverage report. You can also install an additional pytest-cov extension to provide extra features.

pytest --cov=<package>

The report goes to the terminal on standard output. In the case of this project the command is pytest --cov=fab and the output will look something like this:

$ pytest --cov=fab                
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: .../fab
plugins: cov-2.8.1
collected 17 items                                                             

unit-tests/database_test.py ..                                           [ 11%]
unit-tests/reader_test.py .....                                          [ 41%]
unit-tests/source_tree_test.py ...                                       [ 58%]
unit-tests/language/fortran_test.py .......                              [100%]

----------- coverage: platform linux, python 3.6.8-final-0 -----------
Name                              Stmts   Miss  Cover
-----------------------------------------------------
source/fab/__init__.py                2      0   100%
source/fab/__main__.py               29     29     0%
source/fab/application.py            51     51     0%
source/fab/database.py               72      3    96%
source/fab/entry.py                  48     48     0%
source/fab/language/__init__.py      42      6    86%
source/fab/language/fortran.py      204     13    94%
source/fab/queue.py                   4      4     0%
source/fab/reader.py                 47      2    96%
source/fab/source_tree.py            49      3    94%
-----------------------------------------------------
TOTAL                               548    159    71%


============================== 17 passed in 0.92s ==============================

Understanding the Output

The first column to consider is the last. This shows the percentage coverage. Any file which has 100% coverage is good to go and needn't be considered any further.

The remaining two columns show the number of Python statements in the file and the number which were not executed by any test. It is from these two values that the percentage is calculated but they don't do a lot more for us.

If you launch the tool using pytest --cov=feb --cov-report=html it will produce a little web site in the directoryhtmlcov. This will present the same summary table in its index page but you can now clock on any file to show an annotated copy of the source. Lines which have not been covered by a test will be highlighted in red. Lines which have been executed at least one will have a green bar to their left.