From e10211bc2cf842725644668165050fa38c75f23e Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Mon, 12 Jul 2021 09:55:44 +1000 Subject: [PATCH 01/14] First iteration, 2/3 done. --- src/book/03-guides/04-general/04-writing-tests.mdx | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 src/book/03-guides/04-general/04-writing-tests.mdx diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx new file mode 100644 index 000000000..e69de29bb From e01247c27f3c57b00edfae2ca8a88c630032fb55 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Mon, 12 Jul 2021 09:55:56 +1000 Subject: [PATCH 02/14] Adds a couple of glossary definitions --- src/book/03-guides/04-general/03-glossary.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/book/03-guides/04-general/03-glossary.mdx b/src/book/03-guides/04-general/03-glossary.mdx index 1ab65c727..6a7b57cd8 100644 --- a/src/book/03-guides/04-general/03-glossary.mdx +++ b/src/book/03-guides/04-general/03-glossary.mdx @@ -76,6 +76,8 @@ slug: /guides/general/glossary [**GitHub**](https://github.com/): A source code repository hosting service, which uses a git backend. NUbots uses GitHub to manage the source code. +[**GitHub Issue**](https://guides.github.com/features/issues/): A numbered thread associated with a GitHub repository. Issues are commonly used to keep track of bugs, enhancements, or simply hosting discussions. + [**Green Horizon**](/system/subsystems/vision#green-horizon-detector): The edge of the football field, as calculated by the vision system. [**igus Humanoid Open Platform**](https://arxiv.org/abs/1809.11110): The 3d printed base design which the NUgus is based on. @@ -152,6 +154,8 @@ slug: /guides/general/glossary **Quintic Walk**: Open loop walk engine which uses quintic splines to create trajectories. Created by Bit-Bots, based off code from team Rhoban, then ported to run with NUClear. +[**Random Seed**](https://en.wikipedia.org/wiki/Random_seed): A number which is used by a (pseudo-)random number generator to start generating random numbers. Providing a seed for a generator makes its subsequent calls consistent from run to run. + [**Rhoban**](https://www.rhoban-project.fr/): The RoboCup team from Bordeaux University, who created the original quintic walk adapted by Bit-Bots, and ported to the NUbots codebase. [**RoboCup Symposium**](https://www.robocup.org/symposium): The research conference which is held after RoboCup each year, featuring research from the teams competing. From dc4cb91e16c4b4c10e93823033d18e00d3c7a0b2 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Mon, 12 Jul 2021 10:03:14 +1000 Subject: [PATCH 03/14] Moves most of the guide into a general testing guide. The catch guide will be more directly about catch --- .../03-guides/02-tools/05-catch-guide.mdx | 12 ++++ .../03-guides/04-general/04-writing-tests.mdx | 60 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 src/book/03-guides/02-tools/05-catch-guide.mdx diff --git a/src/book/03-guides/02-tools/05-catch-guide.mdx b/src/book/03-guides/02-tools/05-catch-guide.mdx new file mode 100644 index 000000000..11abffef1 --- /dev/null +++ b/src/book/03-guides/02-tools/05-catch-guide.mdx @@ -0,0 +1,12 @@ +--- +section: Guides +chapter: General +title: Catch Getting Started +description: How to write your first test with catch. +slug: /guides/general/catch-guide +--- + +[Catch](https://github.com/catchorg/Catch2) is the C++ testing framework which we use for our unit tests. In this guide we will go over the basics of Catch and how to make a unit test. This guide assumes familiarity with C++. + + + diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index e69de29bb..e4083e41d 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -0,0 +1,60 @@ +--- +section: Guides +chapter: Tools +title: How to Write Tests +description: General Information About Making Tests +slug: /guides/tools/catch-tests +--- + +This guide presents some general information about unit testing and tests in general. + +## What is a Unit Test? + +A unit test is a test of a small piece of a codebase - a unit. Unit tests are meant to test only that piece of code and they serve to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function + +```cpp +[[nodiscard]] inline int add(const int a, const int b) noexcept { + return a + b; +} +``` + +## Anatomy of a Unit Test + +The basic pieces you will need to create a unit test for a function are the following: + +1. A set of inputs for the function. These should match the input parameters. In our case, we'll need a set of pairs of integers. +2. The associated outputs which you want for each input. These are often referred to as the "ground truth", because they are the true values your function should output. For our `add` example with the pair of inputs (2, 2), we want the output 4, because `add(2, 2) == 4`. + +The process of running the tests is as simple as calling the function with your inputs and verifying that the function's output was as expected. In our example, one of the test cases could be running `add(2, 2)` and checking that the result is 4. + +## Testing Approach and Philosophy + +There are many guidelines you can find online to help you to write good tests. Here are a few: + +- Tests should be simple and readable enough to be correct on inspection. You don't want to think about whether a test is correct or not. Ideally you'll be able to read it and know that it's legitimate. +- Make test cases independent. The outcome of one test case shouldn't affect the outcome of others. +- Demonstrate how a piece of code should be used with its tests. We can't google for examples of people using our software, so create examples with your tests. +- Tests should be deterministic - [seed your randomness](https://en.wikipedia.org/wiki/Random_seed). If you're testing something particularly reliant on randomness or which generates randomness, compensate by using a variety of seeds with many cases each. +- Follow the **AAA** structure: Arrange, Act, Assert. Each test should follow the general design of first setting up your input variables (Arrange), calling your unit with those variables (Act), then finally checking your outputs match what they should (Assert). + +### General Approach + +Write the easy tests first, then think about edge cases and code coverage. For `add`, you might make the `(2, 2) == 4` case first, then `(123000, 456) == 123456`. After you have some simple cases, you could consider throwing in some zeros and negative numbers - cases where the observed behaviour is different somehow. + +Later, you might consider what should happen on integer overflow/wraparound, making sure that errors are handled correctly. At that stage you could also try to make a test case covering every possible branch of your code. The amount of code you execute in a set of tests is referred to as the "code coverage" of the tests. + +### Regression Tests + +When we find bugs in the codebase and fix them, we should add a test case which makes sure that that code doesn't _regress_ into the buggy behaviour. This test case is called a regression test. Regression tests should be labelled with comments to indicate the behaviour they're watching for. If there is a GitHub The regression test should be written such that it would fail before the fix, and pass after the fix. + +For a concrete example, imagine that a bug was found with `add` where when both inputs were negative, it always returned 0. Good practice would be to add a test case or small set of test cases where both inputs were negative, labelling them as regression tests for that bug. + +### Black Box vs White Box Testing + +When we write tests, we can make them completely ignorant to the internals of the code. Such a test worries only about the inputs and outputs, considering the parts in the middle as a black box which we can't see inside - black box testing. + +White box testing looks at the internals and makes tests which depend on them. This means that any significant change to the implementation of a function which doesn't change its interface is likely to break white box tests which depended on the old function. The fragility of white box tests to change is the chief reason that black box tests are preferred. + +Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development), which is a powerful means of creating high quality software. + + From f844658801e7c8398e6ae3a7a99db59b84ed9735 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Mon, 12 Jul 2021 10:16:47 +1000 Subject: [PATCH 04/14] Finishes the general guide (pre-review) --- src/book/03-guides/04-general/04-writing-tests.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index e4083e41d..67f28a1bb 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -1,9 +1,9 @@ --- section: Guides -chapter: Tools +chapter: General title: How to Write Tests -description: General Information About Making Tests -slug: /guides/tools/catch-tests +description: Information About Making Tests +slug: /guides/general/writing-tests --- This guide presents some general information about unit testing and tests in general. @@ -45,9 +45,9 @@ Later, you might consider what should happen on integer overflow/wraparound, mak ### Regression Tests -When we find bugs in the codebase and fix them, we should add a test case which makes sure that that code doesn't _regress_ into the buggy behaviour. This test case is called a regression test. Regression tests should be labelled with comments to indicate the behaviour they're watching for. If there is a GitHub The regression test should be written such that it would fail before the fix, and pass after the fix. +When we find bugs in the codebase and fix them, we should add a test case which makes sure that that code doesn't _regress_ into the buggy behaviour. This test case is called a regression test. Regression tests should be labelled with comments to indicate the behaviour they're watching for. If there is a GitHub issue related to the bug, the comment with the test should reference it. The test should be written such that it would fail before the fix, and pass after the fix. -For a concrete example, imagine that a bug was found with `add` where when both inputs were negative, it always returned 0. Good practice would be to add a test case or small set of test cases where both inputs were negative, labelling them as regression tests for that bug. +For a concrete example, imagine that a bug was found with `add` where if both inputs were negative, it always returned 0. Good practice would be to add a test case or small set of test cases where both inputs were negative - such as `(-1, -1) == -2` and `(-123, -456) == -579` - labelling them as regression tests for that bug. These would clearly fail if the bug came back (although this example is quite contrived, because there should have already been tests with both inputs negative). ### Black Box vs White Box Testing @@ -55,6 +55,6 @@ When we write tests, we can make them completely ignorant to the internals of th White box testing looks at the internals and makes tests which depend on them. This means that any significant change to the implementation of a function which doesn't change its interface is likely to break white box tests which depended on the old function. The fragility of white box tests to change is the chief reason that black box tests are preferred. -Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development), which is a powerful means of creating high quality software. - +Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development)(TDD), which is a powerful means of creating high quality software. Tests written as part of a TDD process should inherently be black box tests, because the implementations they're testing don't exist yet. +Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. \ No newline at end of file From 9fb9fcd6799f086b4ca2220d516d2ac802b16555 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Wed, 14 Jul 2021 07:59:15 +1000 Subject: [PATCH 05/14] Formatting --- src/book/03-guides/04-general/04-writing-tests.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index 67f28a1bb..c2e8f53d3 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -10,7 +10,7 @@ This guide presents some general information about unit testing and tests in gen ## What is a Unit Test? -A unit test is a test of a small piece of a codebase - a unit. Unit tests are meant to test only that piece of code and they serve to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function +A unit test is a test of a small piece of a codebase - a unit. Unit tests are meant to test only that piece of code and they serve to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function ```cpp [[nodiscard]] inline int add(const int a, const int b) noexcept { @@ -39,7 +39,7 @@ There are many guidelines you can find online to help you to write good tests. H ### General Approach -Write the easy tests first, then think about edge cases and code coverage. For `add`, you might make the `(2, 2) == 4` case first, then `(123000, 456) == 123456`. After you have some simple cases, you could consider throwing in some zeros and negative numbers - cases where the observed behaviour is different somehow. +Write the easy tests first, then think about edge cases and code coverage. For `add`, you might make the `(2, 2) == 4` case first, then `(123000, 456) == 123456`. After you have some simple cases, you could consider throwing in some zeros and negative numbers - cases where the observed behaviour is different somehow. Later, you might consider what should happen on integer overflow/wraparound, making sure that errors are handled correctly. At that stage you could also try to make a test case covering every possible branch of your code. The amount of code you execute in a set of tests is referred to as the "code coverage" of the tests. @@ -57,4 +57,4 @@ White box testing looks at the internals and makes tests which depend on them. T Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development)(TDD), which is a powerful means of creating high quality software. Tests written as part of a TDD process should inherently be black box tests, because the implementations they're testing don't exist yet. -Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. \ No newline at end of file +Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. From 140486b70934cf9ea9738521647971a448ffa834 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Wed, 14 Jul 2021 08:50:55 +1000 Subject: [PATCH 06/14] Cooks up a basic catch example --- .../03-guides/02-tools/05-catch-guide.mdx | 59 ++++++++++++++++++- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/src/book/03-guides/02-tools/05-catch-guide.mdx b/src/book/03-guides/02-tools/05-catch-guide.mdx index 11abffef1..663c5eebf 100644 --- a/src/book/03-guides/02-tools/05-catch-guide.mdx +++ b/src/book/03-guides/02-tools/05-catch-guide.mdx @@ -1,12 +1,65 @@ --- section: Guides -chapter: General +chapter: Tools title: Catch Getting Started description: How to write your first test with catch. -slug: /guides/general/catch-guide +slug: /guides/tools/catch-guide --- -[Catch](https://github.com/catchorg/Catch2) is the C++ testing framework which we use for our unit tests. In this guide we will go over the basics of Catch and how to make a unit test. This guide assumes familiarity with C++. +[Catch](https://github.com/catchorg/Catch2) is the C++ testing framework which we use for our unit tests. In this guide we will go over the basics of Catch by completing a concrete example. This guide assumes familiarity with C++ and testing concepts. To brush-up on testing, see the [Guide for Writing Tests](/guides/general/writing-tests). +Note that the [Catch docs](https://github.com/catchorg/Catch2/tree/devel/docs) should be the first thing to look at if you're wondering about a specific Catch-ism. +## A Basic Example +Catch test cases have a few components. The most important is the `TEST_CASE(..)` macro, which wraps around groups of associated tests. Inside each `TEST_CASE` scope, you should follow the **AAA** structure, _Arranging_ the data first - both inputs and expected outputs - then _Acting_ by calling the function you're testing, then _Asserting_ that the results match the ground truth. We'll make an example for the following toy utility function: + +```cpp +[[nodiscard]] inline int add(const int a, const int b) noexcept { + return a + b; +} +``` + +To test it, we'll need pairs of inputs and their associated outputs. Usually, if there's a lot of data, we'll want to keep the it separate from the test logic, but for this example we'll keep it local. Also note that for utilities like this one, we would want a much more comprehensive set of test cases. + +```cpp +using utility::math::add; + +TEST_CASE("Testing integer add utility", "[utility][math][add]") { + // Arrange + static constexpr ssize_t NUM_TESTS = 5; + std::array, NUM_TESTS> inputs = {{0, 0}, {1, 1}, {-1, -1}, {123000, 456}, {-1000, 1000}}; + std::array ground_truth = {0, 2, -2, 123456, 0}; + std::array outputs{}; + // Act + for (ssize_t i = 0; i < NUM_TESTS; ++i) { + outputs[i] = add(inputs[i].first, inputs[i].second); + } + // Assert + for (ssize_t i = 0; i < NUM_TESTS; ++i) { + INFO("In test case number " << i); + INFO("Inputs are (" << inputs[i].first << ", " << inputs[i].second << ")"); + INFO("Ground truth is " << ground_truth[i]); + INFO("Function output is " << outputs[i]); + REQUIRE(outputs[i] == ground_truth[i]); + } +} +``` + +### Dissecting the Example + +As we can see in this example, inside the scope of the `TEST_CASE` is where it all happens. We use `INFO` macros to commentate exactly what is happening for each assertion. You shouldn't worry too much about creating huge, unwieldy logs with `INFO` macros, because Catch only prints the `INFO` for test cases which fail by default. + +The first argument for `TEST_CASE` is a string which is a name for the test. You can use the names to run the specific test. They should be specific. The second argument is a set of tags, which you can use to divide the tests into groups easily. We usually use each sub-namespace the function is located in as the tags. The Catch `TEST_CASE` has much more functionality than demonstrated here and you can find the documentation with the details [here](https://github.com/catchorg/Catch2/blob/devel/docs/test-cases-and-sections.md). + +The rest of the example is just going through the **AAA** process (the comments are for illustrative purposes). + +## Floating Point Considerations + +Floating point arithmetic is imprecise by nature. Equality comparisons between distinct non-zero floating point numbers are assumed to be false because of this imprecision. Catch has features to deal with the error from floating point operations. We recommended that if you're testing functions which compute floating point numbers that you [read about those features](https://github.com/catchorg/Catch2/blob/devel/docs/assertions.md#floating-point-comparisons). + +Basically, you'll need to define a margin of error - either relative or absolute - that you can tolerate and use that to define your `Approx` for each floating point `REQUIRE` assertion. + +## Conclusions + +Catch is concise and powerful. It has many more features than the basic example presented here - this is just enough to make you dangerous. Now go and write some tests! From a42afbe4d1770c6a8b49cb4b54f637db6682c405 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Wed, 14 Jul 2021 08:51:08 +1000 Subject: [PATCH 07/14] Adds stuff about Error to the glossary --- src/book/03-guides/04-general/03-glossary.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/book/03-guides/04-general/03-glossary.mdx b/src/book/03-guides/04-general/03-glossary.mdx index 6a7b57cd8..7bd9ffa4e 100644 --- a/src/book/03-guides/04-general/03-glossary.mdx +++ b/src/book/03-guides/04-general/03-glossary.mdx @@ -22,6 +22,8 @@ slug: /guides/general/glossary [**Affine Transform**](https://en.wikipedia.org/wiki/Affine_transformation): A transform which keeps straight lines straight, and parallel lines parallel, without preserving distances or angles between lines otherwise. +[**Approximation Error**](https://en.wikipedia.org/wiki/Approximation_error): The difference between a numerical approximation and the actual value being approximated. It can be expressed as either a relative error term, which is a ratio of error:actual_value, or an absolute error term which is just the error, without reference to the actual value. + [**Arch**](https://www.archlinux.org/): A lightweight linux distribution which the NUgus robots use as their primary operating system. [**Armadillo (arma)**](http://arma.sourceforge.net/): The C++ linear algebra library developed by the CSIRO, previously used by NUbots. Documentation can be found [here](http://arma.sourceforge.net/docs.html). From 37a30b1e7641684503771a4b4da47e0709ad360d Mon Sep 17 00:00:00 2001 From: Kip Hamiltons <48076495+KipHamiltons@users.noreply.github.com> Date: Sat, 14 Aug 2021 16:55:27 +1000 Subject: [PATCH 08/14] Apply suggestions from code review Co-authored-by: Ysobel Sims <35280100+ysims@users.noreply.github.com> --- src/book/03-guides/04-general/04-writing-tests.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index c2e8f53d3..cffbe6563 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -10,7 +10,7 @@ This guide presents some general information about unit testing and tests in gen ## What is a Unit Test? -A unit test is a test of a small piece of a codebase - a unit. Unit tests are meant to test only that piece of code and they serve to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function +A unit test is a test of a small piece of a codebase - a unit. Unit tests should test a single piece of code to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function ```cpp [[nodiscard]] inline int add(const int a, const int b) noexcept { @@ -55,6 +55,6 @@ When we write tests, we can make them completely ignorant to the internals of th White box testing looks at the internals and makes tests which depend on them. This means that any significant change to the implementation of a function which doesn't change its interface is likely to break white box tests which depended on the old function. The fragility of white box tests to change is the chief reason that black box tests are preferred. -Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development)(TDD), which is a powerful means of creating high quality software. Tests written as part of a TDD process should inherently be black box tests, because the implementations they're testing don't exist yet. +Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development) (TDD), which is a powerful means of creating high quality software. Tests written as part of a TDD process should inherently be black box tests, because the implementations they're testing don't exist yet. Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. From 6b9ef0f6bbc21a0b49c9e588d1ea60b4f6173085 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 16:58:36 +1000 Subject: [PATCH 09/14] Strip down add function example to be super simple --- src/book/03-guides/04-general/04-writing-tests.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index cffbe6563..b2dbac9bf 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -13,7 +13,7 @@ This guide presents some general information about unit testing and tests in gen A unit test is a test of a small piece of a codebase - a unit. Unit tests should test a single piece of code to validate its correctness. This contrasts with integration tests, which tests the interactions of pieces of code and their behaviour together. For the rest of this guide, we'll work with the following toy utility function ```cpp -[[nodiscard]] inline int add(const int a, const int b) noexcept { +int add(int a, int b) { return a + b; } ``` From 40c52fdb32f1315ccd8bddd69bab3e0b7849c309 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 16:59:55 +1000 Subject: [PATCH 10/14] Updates the add function in the catch guide too. --- src/book/03-guides/02-tools/05-catch-guide.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/book/03-guides/02-tools/05-catch-guide.mdx b/src/book/03-guides/02-tools/05-catch-guide.mdx index 663c5eebf..4b4cbf0d0 100644 --- a/src/book/03-guides/02-tools/05-catch-guide.mdx +++ b/src/book/03-guides/02-tools/05-catch-guide.mdx @@ -15,7 +15,7 @@ Note that the [Catch docs](https://github.com/catchorg/Catch2/tree/devel/docs) s Catch test cases have a few components. The most important is the `TEST_CASE(..)` macro, which wraps around groups of associated tests. Inside each `TEST_CASE` scope, you should follow the **AAA** structure, _Arranging_ the data first - both inputs and expected outputs - then _Acting_ by calling the function you're testing, then _Asserting_ that the results match the ground truth. We'll make an example for the following toy utility function: ```cpp -[[nodiscard]] inline int add(const int a, const int b) noexcept { +int add(int a, int b) { return a + b; } ``` From 214836b19e09c447513791801d70812b743e9252 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 17:03:05 +1000 Subject: [PATCH 11/14] Chooses "expected output" over "ground truth" --- .../03-guides/02-tools/05-catch-guide.mdx | 20 +++++++++---------- .../03-guides/04-general/04-writing-tests.mdx | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/book/03-guides/02-tools/05-catch-guide.mdx b/src/book/03-guides/02-tools/05-catch-guide.mdx index 4b4cbf0d0..1e716b15f 100644 --- a/src/book/03-guides/02-tools/05-catch-guide.mdx +++ b/src/book/03-guides/02-tools/05-catch-guide.mdx @@ -12,7 +12,7 @@ Note that the [Catch docs](https://github.com/catchorg/Catch2/tree/devel/docs) s ## A Basic Example -Catch test cases have a few components. The most important is the `TEST_CASE(..)` macro, which wraps around groups of associated tests. Inside each `TEST_CASE` scope, you should follow the **AAA** structure, _Arranging_ the data first - both inputs and expected outputs - then _Acting_ by calling the function you're testing, then _Asserting_ that the results match the ground truth. We'll make an example for the following toy utility function: +Catch test cases have a few components. The most important is the `TEST_CASE(..)` macro, which wraps around groups of associated tests. Inside each `TEST_CASE` scope, you should follow the **AAA** structure, _Arranging_ the data first - both inputs and expected outputs - then _Acting_ by calling the function you're testing, then _Asserting_ that the results match the expected outputs. We'll make an example for the following toy utility function: ```cpp int add(int a, int b) { @@ -27,21 +27,21 @@ using utility::math::add; TEST_CASE("Testing integer add utility", "[utility][math][add]") { // Arrange - static constexpr ssize_t NUM_TESTS = 5; + static constexpr int NUM_TESTS = 5; std::array, NUM_TESTS> inputs = {{0, 0}, {1, 1}, {-1, -1}, {123000, 456}, {-1000, 1000}}; - std::array ground_truth = {0, 2, -2, 123456, 0}; - std::array outputs{}; + std::array expected_outputs = {0, 2, -2, 123456, 0}; + std::array actual_outputs{}; // Act - for (ssize_t i = 0; i < NUM_TESTS; ++i) { - outputs[i] = add(inputs[i].first, inputs[i].second); + for (int i = 0; i < NUM_TESTS; ++i) { + actual_outputs[i] = add(inputs[i].first, inputs[i].second); } // Assert - for (ssize_t i = 0; i < NUM_TESTS; ++i) { + for (int i = 0; i < NUM_TESTS; ++i) { INFO("In test case number " << i); INFO("Inputs are (" << inputs[i].first << ", " << inputs[i].second << ")"); - INFO("Ground truth is " << ground_truth[i]); - INFO("Function output is " << outputs[i]); - REQUIRE(outputs[i] == ground_truth[i]); + INFO("Expected output is " << expected_outputs[i]); + INFO("Actual output is " << actual_outputs[i]); + REQUIRE(actual_outputs[i] == expected_outputs[i]); } } ``` diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index b2dbac9bf..d22b753b4 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -23,7 +23,7 @@ int add(int a, int b) { The basic pieces you will need to create a unit test for a function are the following: 1. A set of inputs for the function. These should match the input parameters. In our case, we'll need a set of pairs of integers. -2. The associated outputs which you want for each input. These are often referred to as the "ground truth", because they are the true values your function should output. For our `add` example with the pair of inputs (2, 2), we want the output 4, because `add(2, 2) == 4`. +2. The associated outputs which you want for each input. These expected outputs are often referred to as the "ground truth", because they are the true values your function should output. For our `add` example with the pair of inputs (2, 2), we want the output 4, because `add(2, 2) == 4`. The process of running the tests is as simple as calling the function with your inputs and verifying that the function's output was as expected. In our example, one of the test cases could be running `add(2, 2)` and checking that the result is 4. From 7028b3f4d8627ee4e16d1ed40bb8e0621ba0c0ed Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 17:31:28 +1000 Subject: [PATCH 12/14] Adds more suggestions from review --- src/book/03-guides/02-tools/05-catch-guide.mdx | 4 ++-- src/book/03-guides/04-general/04-writing-tests.mdx | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/book/03-guides/02-tools/05-catch-guide.mdx b/src/book/03-guides/02-tools/05-catch-guide.mdx index 1e716b15f..f507246a0 100644 --- a/src/book/03-guides/02-tools/05-catch-guide.mdx +++ b/src/book/03-guides/02-tools/05-catch-guide.mdx @@ -20,7 +20,7 @@ int add(int a, int b) { } ``` -To test it, we'll need pairs of inputs and their associated outputs. Usually, if there's a lot of data, we'll want to keep the it separate from the test logic, but for this example we'll keep it local. Also note that for utilities like this one, we would want a much more comprehensive set of test cases. +To test it, we'll need pairs of inputs and their expected outputs. Usually, if there's a lot of data, we'll want to keep it separate from the test logic, but for this example we'll keep it local. Also note that for utilities like this one, we would want a much more comprehensive set of test cases. ```cpp using utility::math::add; @@ -48,7 +48,7 @@ TEST_CASE("Testing integer add utility", "[utility][math][add]") { ### Dissecting the Example -As we can see in this example, inside the scope of the `TEST_CASE` is where it all happens. We use `INFO` macros to commentate exactly what is happening for each assertion. You shouldn't worry too much about creating huge, unwieldy logs with `INFO` macros, because Catch only prints the `INFO` for test cases which fail by default. +As we can see in this example, inside the scope of the `TEST_CASE` is where it all happens. We use `INFO` macros to commentate exactly what is happening for each assertion. You shouldn't worry too much about creating huge, unwieldy logs with `INFO` macros, because by default Catch only prints the `INFO` for test cases which fail. The first argument for `TEST_CASE` is a string which is a name for the test. You can use the names to run the specific test. They should be specific. The second argument is a set of tags, which you can use to divide the tests into groups easily. We usually use each sub-namespace the function is located in as the tags. The Catch `TEST_CASE` has much more functionality than demonstrated here and you can find the documentation with the details [here](https://github.com/catchorg/Catch2/blob/devel/docs/test-cases-and-sections.md). diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index d22b753b4..c337ac15f 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -23,7 +23,7 @@ int add(int a, int b) { The basic pieces you will need to create a unit test for a function are the following: 1. A set of inputs for the function. These should match the input parameters. In our case, we'll need a set of pairs of integers. -2. The associated outputs which you want for each input. These expected outputs are often referred to as the "ground truth", because they are the true values your function should output. For our `add` example with the pair of inputs (2, 2), we want the output 4, because `add(2, 2) == 4`. +2. The expected outputs which you want for each input. These are often referred to as the "ground truth", because they are the true values your function should output. For our `add` example with the pair of inputs (2, 2), we want the output 4, because `add(2, 2) == 4`. The process of running the tests is as simple as calling the function with your inputs and verifying that the function's output was as expected. In our example, one of the test cases could be running `add(2, 2)` and checking that the result is 4. From dedb84d9c74184822f0fdb4294c7e9481b9c9d16 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 17:31:49 +1000 Subject: [PATCH 13/14] Adds section about generating data Thanks for the suggestion Liam --- src/book/03-guides/04-general/04-writing-tests.mdx | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index c337ac15f..553ebe108 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -27,6 +27,14 @@ The basic pieces you will need to create a unit test for a function are the foll The process of running the tests is as simple as calling the function with your inputs and verifying that the function's output was as expected. In our example, one of the test cases could be running `add(2, 2)` and checking that the result is 4. +### Obtaining Expected Outputs + +For our C++ code, we can divide the things we want to test into two categories. The first category is the set of functions which do a mathematical transformation on the data and return the result. The second is everything else. Functions which just do maths - which includes things like filters - often require large sets of generated data or randomised inputs across their domain. When we generate data like this, we must document how we generated it and how we verified its correctness. To verify its correctness, you can simply use an implementation for a different platform or language that you're sure is correct. For example, if we were testing our `add` function, we could randomly generate a list of inputs, making sure that each corner case is covered, then use `numpy.add` on each input to verify the expected outputs. We can be quite sure that the `numpy` implementation is correct, so all we would have to do is make it clear that that was the process we used. + +Functions which aren't just maths will have more "categorical" corner cases and input domains. For such functions, data generation or randomised inputs are rarely appropriate. We should identify and cover each of those cases with tests manually. Note that the simple cases should have tests too. Don't just test edge cases. + +In general, for data generation we want to use languages and libraries that we already use regularly. Python is most often preferred for this, but Matlab is acceptable if it has an implementation of something python lacks. Other languages should be avoided where possible. + ## Testing Approach and Philosophy There are many guidelines you can find online to help you to write good tests. Here are a few: From 1e968eb76b126d76d32f1bbc321ae30a630934a7 Mon Sep 17 00:00:00 2001 From: Kip Hamiltons Date: Sat, 14 Aug 2021 17:57:57 +1000 Subject: [PATCH 14/14] Adds BDD/TDD section --- src/book/03-guides/04-general/04-writing-tests.mdx | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/book/03-guides/04-general/04-writing-tests.mdx b/src/book/03-guides/04-general/04-writing-tests.mdx index 553ebe108..85d7aac44 100644 --- a/src/book/03-guides/04-general/04-writing-tests.mdx +++ b/src/book/03-guides/04-general/04-writing-tests.mdx @@ -63,6 +63,16 @@ When we write tests, we can make them completely ignorant to the internals of th White box testing looks at the internals and makes tests which depend on them. This means that any significant change to the implementation of a function which doesn't change its interface is likely to break white box tests which depended on the old function. The fragility of white box tests to change is the chief reason that black box tests are preferred. +Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. + +## TDD and BDD + Ideally, you're able to write tests as you design your software interfaces, writing the code afterwards such that it fulfills the needs of the tests. This is the basis of [**Test-Driven Development**](https://en.wikipedia.org/wiki/Test-driven_development) (TDD), which is a powerful means of creating high quality software. Tests written as part of a TDD process should inherently be black box tests, because the implementations they're testing don't exist yet. -Grey box testing is somewhere between black box and white box testing. It isn't completely ignorant of the implementation, but grey box tests should be easily adaptable if the implementation changes. It will usually be necessary to have some sort of insight into the implementation of the unit in order to get full code coverage, but a rule of thumb is the blacker the box, the better. +Another conception of TDD prescribes writing the tests in parallel with the code. With this process, you write a basic test in conjunction with starting a new module or feature. As you write the code, you increase the number and depth of the tests, developing them in tandem. This incremental style is more in keeping with the [_agile_ philosophy](https://www.atlassian.com/agile). The code works at each step of development and the tests can prove it. We aren't agile die-hards though, so we don't have a particular preference between these two realisations of TDD. + +[Behaviour-driven development](https://en.wikipedia.org/wiki/Behavior-driven_development) (BDD) is a more modern take on TDD. BDD tests can be thought of as self-documenting scripts which should be able to be understood by all stakeholders. It formally redefines the _Arrange, Act, Assert_ structure as _Given, When, Then_. Each BDD test should be of the form _Given_ some situation, _When_ a specific thing happens, _Then_ the system responds in the correct way. Following this language structure predisposes the developer to writing tests which everyone can understand. The Catch docs have [a great example](https://github.com/catchorg/Catch2/blob/devel/docs/test-cases-and-sections.md#bdd-style-test-cases) demonstrating the structure of a BDD test. + +In terms of the ideal development cycle and testing style, it doesn't matter all too much whether you use TDD, "agile style" TDD (not a real name), or BDD. Having a mix of BDD style tests, which are by their nature designed for everyone to be able to understand, and more fine-grained, standard unit tests is good. The most important thing is writing readable, verifiable tests. Everything else is secondary. + +So what are you waiting for? Go write some tests!