Unit test coverage

[JannesBrands]

I'm reworking at the moment some things of @Thom-de-Jong 's object pool reading and writing implementation. Also I already integrated some unit tests. Right now I'm asking myself how much test coverage we want to achieve or if we prefer to focus on a fast progressing library?

Testing all the ISO11783-6 objects for reading and writing takes a lot of time, so I asked myself what our general implementation strategy is.
@GwnDaan
@Notgnoshi
@Thom-de-Jong
@ad3154

[Notgnoshi]

Great question! Boy do I have opinions 😆

(high value) tests are a huge asset to an open source project. But I think we should also be pragmatic - don't target a magical code coverage number - test what needs to be tested.

Here's my reasoning

I've worked on projects that have taken it too far, and those who haven't taken it far enough. I think automated testing provides a few values (in order according to my opinion):

1. In a community project, enables non-intimately-familiar-maintainers to contribute!!
2. Push the developer in a better direction architecturally - writing good tests means you have to decompose the code under test, and think about how the user is going to use it
3. Documentation of developer expectations - how many times have I written code expecting it would be used one way, and then when it's used some other way it explodes?
4. Enforce "correctness" - maybe better stated as "prevent regressions". I've seen a lot of tests that have asserted incorrect behavior :(

But tests can also be a liability:

• I've seen lots of tests that provide negative value, because they don't assert correctness, they assert tautologies
• I've seen developers come up with insane architectures to make something that wasn't designed to be testable, testable
• I've seen open source projects have aggressive testing policies that discourage community contributions

So, I think we should encourage testing (especially as we really want community contribution)! But we also shouldn't go overboard by requiring code coverage percentages, or using complex mocking frameworks that are more complex than the code under test :)

[GwnDaan]

Hi, I totally agree with your reasoning. And I think we all agree that automated tests are necessary to maintain good quality of the code base, though I still wonder how this translates in practice. When do we define something as "should be tested"? I think it's still really subjective right now.

Let's take the object pool encoding/decoding tests @JannesBrands was writing as an example, by reading your thoughts on unit test I expect you will value them as a good asset but I'm not 100% sure. Feel free to correct me. If do want tests for this, we also have to consider to which extend does such a test go:

• does one need to test only one complete pool of objects, or is it better to test each object individually and only do some general broader checks on the complete pool?
• does one need to test correctness for multiple variations of an object (i.e. edge cases), or is the base enough?
• where could one be testing a tautologie instead of correctness for this particular case? Would that be testing the encoding based on the result of the decoding?

In general, I think this really is a valuable discussion. And encouring, instead of enforcing is quite important, I feel like in most cases bad tests are worse than no tests for maintainability.

[JannesBrands]

does one need to test only one complete pool of objects, or is it better to test each object individually and only do some general broader checks on the complete pool?

I would say we focus on unit tests. The unit test I created was not really appropriate. Due to the handling of the read/write function of @Thom-de-Jong that process all the object types inside one function was the reason why my unit test was way too big. So I started already refactoring it and finished it yesterday 980b38a (and following commits in #27)
I suggest not creating large unit tests or considering whether refactoring is necessary.

does one need to test correctness for multiple variations of an object (i.e. edge cases), or is the base enough?

The same I asked myself for my unit tests. Right now im just parsing an example pool. So just the cases of the example pool are proven. I would prefer to test all cases but that would take a lot time, that is why I created this discussion.