diff --git a/apps/cookbook/docs/angular/testing/01-beyond-unit-vs-integration.mdx b/apps/cookbook/docs/angular/01-testing/01-beyond-unit-vs-integration/index.mdx similarity index 75% rename from apps/cookbook/docs/angular/testing/01-beyond-unit-vs-integration.mdx rename to apps/cookbook/docs/angular/01-testing/01-beyond-unit-vs-integration/index.mdx index 8b7f6c6..bfacbf8 100644 --- a/apps/cookbook/docs/angular/testing/01-beyond-unit-vs-integration.mdx +++ b/apps/cookbook/docs/angular/01-testing/01-beyond-unit-vs-integration/index.mdx @@ -20,7 +20,7 @@ First of all, there is no universally accepted definition for these categories. - Is it still a unit test **if it uses a database**? - Is it an **integration test** or a **sociable unit test** if it reuses something already tested instead of replacing it with a test double? -The line between unit and integration tests is blurry. While we could might be tempting to accept these ambiguities, it is important to understand that the distinction between these categories goes beyond mere semantics. +The line between unit and integration tests is blurry. While it might be tempting to accept these ambiguities, it is important to understand that the distinction between these categories goes beyond mere semantics. **This confusion is harmful**. It fuels dogmatic testing approaches and leads teams to adopt suboptimal strategies that slow them down or even discourage them from writing tests. @@ -44,17 +44,17 @@ That is why I suggest using the following categories: The Narrow vs. Wide categories resonated more effectively with the teams I’ve had the privilege of coaching over the past few years. ::: -## 🐜 Narrow Tests +## 🐜 Narrow Tests {#narrow-tests} Narrow tests are defined by the following properties: -### ⚡️ Fast +### ⚡️ Fast {#fast} -The goal of Narrow tests is to provide the **fastest feedback time while maintaining the highest level of confidence** possible. This is crucial in to maintain a high development speed. +The goal of Narrow tests is to provide the **fastest feedback time while maintaining the highest level of confidence** possible. This is crucial to maintaining a high development speed. The reasoning behind this is simple: -- 🐇 If the feedback time is **fast enough** _(less than few seconds)_, you are more likely to make **baby steps and see the impact of your changes immediately**, then continue or fix the last change. +- 🐇 If the feedback time is **fast enough** _(less than a few seconds)_, you are more likely to make **baby steps and see the impact of your changes immediately**, then continue or fix the last change. - 🐢 If the feedback time is **too long**, you will run the tests less often, make bigger changes, then spend minutes or hours debugging the failing tests. ![Test Feedback Time Incidence](./test-feedback-time.png) @@ -90,7 +90,7 @@ Some Narrow tests might be a bit slower and it is okay. _(e.g. the first test in Some complex parts of your code might require thousands of Narrow tests, so you will want to reduce the threshold to something like `~10ms`. ::: -### 🏝️ Easy to Isolate and Parallelize +### 🏝️ Easy to Isolate and Parallelize {#easy-to-isolate-and-parallelize} Narrow tests should be **isolated** and thus parallelizable **without @@ -106,11 +106,11 @@ Also, if tests are not isolated, they can't be parallelized, thus slow. Ideally, **even Wide tests should be isolated and parallelizable**. The main difference is that Wide test isolation can be relatively hard to achieve. e.g. you might need to create a distinct "user" or "workspace" per test, or some uniqueness constraints might force you to forge your testing data differently. ::: -If a group of tests are using a **shared resource that requires substantial effort** to avoid side effects between tests, **then these tests are Wide tests**. +If a group of tests is using a **shared resource that requires substantial effort** to avoid side effects between tests, **then these tests are Wide tests**. #### Example: Is it still a Narrow test if it is using a Database? -If each test can have a distinct database populated with the necessary data without breaking the other properties: [Fast](#️-fast) & [Low Cognitive Load](#-low-cognitive-load), then it is a Narrow test. +If each test can have a distinct database populated with the necessary data without breaking the other properties: [Fast](#fast) & [Low Cognitive Load](#low-cognitive-load), then it is a Narrow test. For instance, it takes zero setup and [one line of code](https://github.com/typegoose/mongodb-memory-server/?tab=readme-ov-file#simple-server-start) to create an [In-Memory MongoDB Server](https://github.com/typegoose/mongodb-memory-server). Each worker could easily create its own MongoDB Server _(that starts in less than 500ms)_ then each test can create and populate a distinct database on this server in a few milliseconds. @@ -120,7 +120,7 @@ On the other hand, **if isolation and parallelization are harder to achieve, the - Creating a database server requires more effort (e.g. [test containers](https://testcontainers.com)), or more time to spin up. - Populating the database requires more time. -### 😌 Low Cognitive Load +### 😌 Low Cognitive Load {#low-cognitive-load} While speed and parallelization are vital properties, **the true cornerstone of Narrow tests is Low Cognitive Load**. This is the property that will help you define the boundaries of the System Under Test _(SUT)_. @@ -144,7 +144,7 @@ It is important to note that a smaller SUT does not necessarily mean a lower cog Narrowing tests too much can lead to a high cognitive load due to the complexity of the test setup and test doubles orchestration. ::: -## 🐘 Wide Tests +## 🐘 Wide Tests {#wide-tests} Wide tests are tests that are missing one of the properties of Narrow tests. @@ -156,14 +156,31 @@ They are **either**: Given Wide test properties, one might think that they should simply be avoided. However, Wide tests have their own advantages: -- They are more symmetric to production, and [predictive](https://testdesiderata.com/#:~:text=Predictive), thus more reassuring. -- They are more [structure-insensitive](https://testdesiderata.com/#:~:text=Structure%2Dinsensitive) _(e.g. you can refactor your code without breaking them)_. +- They are more [symmetric to production](../../02-glossary.md#symmetric-to-production), and [predictive](../../02-glossary.md#predictive), thus more reassuring. +- They are more [structure-insensitive](../../02-glossary.md#structure-insensitive) _(e.g. you can refactor your code without breaking them)_. + +## ⚖️ Comparing Narrow and Wide Tests + +| Property | Narrow Tests | Wide Tests | +| ----------------------------------------------------------------------- | --------------------- | --------------------- | +| **[Speed](#fast)** | ⚡️ Fast | 🐢 Slow | +| **[Isolation](#easy-to-isolate-and-parallelize)** | 🏝️ Easy to isolate | 🔗 Harder to isolate | +| **[Cognitive Load](#low-cognitive-load)** | 😌 Low cognitive load | 🤯 Harder to diagnose | +| **[Precision](../../02-glossary.md#precise-tests)** | Higher Precision | Lower Precision | +| **[Production-Symmetry](../../02-glossary.md#symmetric-to-production)** | Lower Symmetry | Higher Symmetry | +| **[Structure-Sensitivity](../../02-glossary.md#structure-insensitive)** | Higher Sensitivity | Lower Sensitivity | + +:::tip Narrow Tests Foster Good Design +It is easier to write Narrow tests for well-designed code with clear separation of concerns, and the right abstractions where we can easily replace dependencies with test doubles. Therefore **Narrow tests are more likely to foster good design**. + +On the other hand, Over-Narrow tests can lead to [over-specification](../../02-glossary.md#over-specification) and make tests more [structure-sensitive](../../02-glossary.md#structure-insensitive) and brittle. +::: ## Which one to choose? A sane testing strategy will have to balance between both categories. -We will discuss this in more detail in a future chapter. +Cf. [Designing a Pragmatic Testing Strategy](../02-designing-a-pragmatic-testing-strategy/index.mdx) ## Additional Resources diff --git a/apps/cookbook/docs/angular/testing/test-feedback-time.png b/apps/cookbook/docs/angular/01-testing/01-beyond-unit-vs-integration/test-feedback-time.png similarity index 100% rename from apps/cookbook/docs/angular/testing/test-feedback-time.png rename to apps/cookbook/docs/angular/01-testing/01-beyond-unit-vs-integration/test-feedback-time.png diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-1.jpg b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-1.jpg new file mode 100644 index 0000000..fb022b7 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-1.jpg differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-2.jpg b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-2.jpg new file mode 100644 index 0000000..1957602 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-2.jpg differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-3.jpg b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-3.jpg new file mode 100644 index 0000000..005b436 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-3.jpg differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-4.jpg b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-4.jpg new file mode 100644 index 0000000..d330092 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-4.jpg differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-5.jpg b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-5.jpg new file mode 100644 index 0000000..417a8fa Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/adapting-5.jpg differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/development-time-perception-bias.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/development-time-perception-bias.png new file mode 100644 index 0000000..d5d2c8b Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/development-time-perception-bias.png differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/honeycomb-test-model.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/honeycomb-test-model.png new file mode 100644 index 0000000..53607a8 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/honeycomb-test-model.png differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/ice-cream-cone-model.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/ice-cream-cone-model.png new file mode 100644 index 0000000..64dae7d Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/ice-cream-cone-model.png differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/index.mdx b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/index.mdx new file mode 100644 index 0000000..9ed137f --- /dev/null +++ b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/index.mdx @@ -0,0 +1,305 @@ +--- +title: Designing a Testing Strategy +slug: /angular/pragmatic-testing-strategy +toc_max_heading_level: 4 +--- + +import { ImageContainer } from '@site/src/components/image-container'; +import { MegaQuote } from '@site/src/components/mega-quote'; + +# Designing a Pragmatic Testing Strategy + +The main goals to keep in mind when designing a testing strategy are: + +- **💪 Confidence**: The tests should provide a high level of confidence to the team that the product works as expected. +- **🪙 Cost**: The tests should be cheap to write and, more importantly, cheap to maintain. Writing tests should not slow down the development process. +- **⚡️ Feedback Speed**: The tests should run fast. If the tests take too long to run, the team will run them less often and will be slowed down in the development process by the feedback loop. + +## Tests Are Safety Nets + +We want to prevent bugs from reaching users, just like a trapeze artist wants to avoid falling to the ground. + +Tests _([Narrow](../../02-glossary.md#narrow-tests) and [Wide](../../02-glossary.md#wide-tests))_ are safety nets among others like [Dogfooding](../../02-glossary.md#dogfooding) or [Canary Release](../../02-glossary.md#canary-release). + +![Safety Nets](./safety-nets.png) + + + It is **cheaper and faster to recover from the first safety net** than the + last one. + + +### The Falling Trapeze Artist + +When a trapeze artist falls, it takes minutes to recover from the first safety net, then the show goes on. But if they fall through the first safety net and hit the landing mat at the bottom of the circus tent _(last "safety net")_, it can take weeks or months to recover. The cost of recovery is also higher _(e.g., audience shock, financial shortfall, medical expenses, etc.)_. + +Needless to say, developing without tests is like performing trapeze without a safety net. It is spectacular, yet reckless. + +## Test Categories Distribution Models + +### Beware the Ice Cream Cone Model + +A common oversight when designing a testing strategy is to mainly focus on [Wide tests](../../02-glossary.md#wide-tests) in general. Things get even worse when the model is topped with a few calories of manual testing. + + + ![Ice Cream Cone Model](./ice-cream-cone-model.png) + + +While such a model could work for a simple product with small ambitions, it **does not scale**. + +The main issues with the Ice Cream Cone Model are: + +- Obviously, manual testing is slow, expensive, and error-prone. _(Humans are bad at repetitive tasks.)_ +- **🐢 Slow Feedback Loop**: Wide tests are slow, and we often have to run them all as it is not always easy to only run [Affected Tests](../../02-glossary.md#affected-tests) for Wide tests. +- **💰 Expensive to Maintain**: Wide tests are attractive as they might sometimes require less setup _(e.g., fewer Test Doubles)_ than Narrow tests. While this can make the initial implementation cheaper, the maintenance cost is generally much more expensive. For instance, Wide tests are not precise by nature and require a higher [cognitive load](../01-beyond-unit-vs-integration/index.mdx#low-cognitive-load). Debugging failing Wide tests is therefore a recurring and expensive task. + +### What Went Wrong with the Test Pyramid? + +The Test Pyramid has been a commonly accepted model for around 20 years. The idea is to focus more on Narrow tests than on Wide tests. + + + ![Test Pyramid](./test-pyramid.png) + + +The Test Pyramid is a great model to follow, but **it is often misunderstood for two reasons**: + +- From the shape of the pyramid, one might think that the narrower the tests are, the better. +- The test pyramid bottom layer is generally defined as "unit tests", but as mentioned in the [previous chapter](../01-beyond-unit-vs-integration/index.mdx), the "unit test" term is confusing and often misinterpreted. + +Due to this frequent misunderstanding, **developers might end up with tests that are too Narrow**. The problem with such tests is that they are: + +- **[Over-Specifying](../../02-glossary.md#over-specification)**: They are too structure-sensitive and coupled to implementation details. They do not survive refactoring. +- **Expensive to Implement**: They require substantial test setup and [test double](../../02-glossary.md#test-doubles) orchestration. +- **Expensive to Maintain**: Being over-specifying makes them expensive to maintain. +- **Not [Predictive](../../02-glossary.md#predictive)**: They are so [asymmetric to production](../../02-glossary.md#symmetric-to-production) that they are not predictive of the system's behavior. All the tests pass, but the feature is actually broken. + +### The Honeycomb Testing Model + +To highlight the fact that Narrow tests should not be too narrow, **prefer the Honeycomb Testing Model**. + + + ![Honeycomb Test Model](./honeycomb-test-model.png) + + + + The goal here is to focus on the widest tests that honor the [Narrow test + definition](../01-beyond-unit-vs-integration/index.mdx#narrow-tests): fast, + isolated & parallelizable, and **low cognitive load**. + + +### Progressive Confidence + +It can be hard to decide how narrow or wide a test should be. Here is another tool that can help you decide: **Progressive Confidence**. + + + ![Progressive Confidence](./progressive-confidence.png) + + + + For most changes, the team should have a relatively high level of confidence + after few seconds to minutes of running the affected Narrow tests. + + +Wider tests and other safety nets like Canary Release are only there to push confidence the extra mile and get as close as possible to ~99% before general availability. + +## When to Narrow Down a Test + +Sometimes, what started as a Narrow test becomes a Wide test. **Think of narrowing down your tests when necessary**. Here are some indicators that it may be time to narrow down your test: + +- the **System Under Test is becoming too complex**, +- the test requires **too many test doubles**, +- or certain parts of the System Under Test are **widely shared across the product**. _(For example, open-source libraries and organization-wide shared libraries require narrower tests.)_ + +## Strengthening the Safety Nets + +Bugs can slip through some of the safety nets. Even if another safety net eventually catches the bug, **it is crucial to improve the earlier safety nets**. This is especially important if bugs went through the same hole in the net more than once. + +For example, if some of your end-to-end tests often catch the same bugs, consider adding some Narrow tests to that specific area. + +## Example: Adapting to the Situation + +For a simple product with small ambitions, one might start with some Wide tests. + +![Adapting to the Situation #1](./adapting-1.jpg) + +This could be fair enough until: + +- some edge cases become hard to cover with Wide tests, +- the tests start relying on external services, making them slow and flaky, +- the tests become hard to isolate and parallelize. + +![Adapting to the Situation #2](./adapting-2.jpg) + +Then, it is time to start zooming in with some Narrow tests. + +![Adapting to the Situation #3](./adapting-3.jpg) + +Sometimes, it will not be enough as some parts of the System Under Test become too complex or too widely shared. + +![Adapting to the Situation #4](./adapting-4.jpg) + +That is when you should consider zooming in even more with narrower tests. + +![Adapting to the Situation #5](./adapting-5.jpg) + +## Composability + +Interestingly, in the example above, we can observe that with such an approach, we can reach a relatively high level of confidence with only a few tests (**1 Wide**, **2 Narrow**, **1 narrower**) and one Test Double. Our tests are [**composable**](../../02-glossary.md#composable). + +The **two extremes that wouldn't work** are the following: + +- **Only Wide Tests** _(ice-cream cone)_: We would probably require dozens of relatively complex Wide tests to reach all the edge cases and the same level of confidence. +- **Over-Narrow Tests** _(misunderstood pyramid)_: We would probably require dozens of over-narrow tests and 3 to 4 Test Doubles to reach all the edge cases. + +## What Should Be Tested? + +Ask future-you! 🫵 + +Put yourself in the shoes of a new maintainer joining your team in 5 weeks... or 5 years. What would you hope was tested to make your contribution smoother and more confident? + +### Risk Assessment + +Another approach is to **compare the cost of testing something to the cost of fixing a related bug when caught by users** _(after piercing through the holes of all the safety nets)_. +Here are some examples: + +- Is it cheaper to test that [AG Grid](https://www.ag-grid.com/angular-data-grid/column-sizing/)'s `resizable` or [Tanstack's Table](https://tanstack.com/table/v8/docs/framework/angular/angular-table#createangulartable)'s `enableColumnResizing` option was properly forwarded to the underlying library, or to fix a bug related to it after a user reports that they cannot resize columns? +- Is it cheaper to test the integration with a Payment API or break the checkout process for a few users... or all users... then spend a few hours debugging and fixing? +- Is it cheaper to test that some rare error case is displaying an actionable error message or to have a customer support agent spend 30 minutes on the phone with each impacted user? + +While you might think that the answer to all these examples is obvious, it is actually subjective. The answer depends on the context of your product, the phase of the product _(Cf. [3X](../../02-glossary.md#3x))_, the team, and the users _(their nature and volume)_, among other things. + +### The Testing Strategy Should Be Part of the Design Doc + +When designing a new feature, the testing strategy for that specific feature should be part of the [Design Doc](../../02-glossary.md#design-doc) _(or whatever drawings or notes you make before coding)_. The testing strategy should be as important as the architecture. + +What if there is no Design Doc? Well, maybe you should give it a try. + +### Development Time Perception Bias + +You might have heard this before: + +> _"We'd love to write tests, but we don't have time now."_ + +:::note +If you don't have time now, it is less likely that you will have more time later -- especially since retrofitting tests is generally more expensive. +::: + +:::warning +Obviously, with the wrong testing strategy _(Cf. [Ice Cream Cone Model](#beware-the-ice-cream-cone-model) or [The Misunderstood Pyramid](#what-went-wrong-with-the-test-pyramid))_, writing tests can slow down the development process. +::: + +However, even with a good testing strategy, the feeling of not having enough time to write tests will probably remain for a while. I call this the **Development Time Perception Bias**: + + + Development Time Perception Bias is the illusion of faster development without + tests due to the permanent action and the heat of the moment _(code, manual + test, debug, fix, ...)_. + + +- **The Aim**: My initial intention is to be a good kid and follow a TDD approach: I want to write a test, see it fail, make it pass, then tidy up. +- **The Attraction**: Then I am drawn to the implementation details by the code mermaids whispering: _"It's just a small change anyway, and you know how to write it properly from the start, right? This way you will not waste time tidying up later. Concerning the test, it's going to be easier to write it once you have the whole implementation done. Generative AI can handle that for you, remember."_ +- **The Reality**: + - I make a change, + - I start the app, + - I wait for it to start, + - oh! what is this new strange unrelated warning in the console? Let's fix it, + - 30 minutes later, where was I? Oh yes, the change, + - I open a browser, + - oh oops, the session expired, I have to authenticate, + - oh two-factor authentication, let's find my phone, + - let's fill this long form again, the component I am working on is the 7th step of the wizard, + - how come the loader is just spinning and nothing is happening? + - let's debug, + - oh, got it, the condition is wrong, let's fix it, + - ... +- **The Perception**: Yay! I am done! That was fast! Wait a minute, why is it dark outside? Where did the day go? Anyway, I couldn't have done it faster, especially if I had to write tests in addition to the feature. After all, the code is not that complex, it does not need tests. + +![Development Time Perception Bias](./development-time-perception-bias.png) + +Testing, and more precisely TDD, **keeps us focused on the task at hand, enabling us to develop faster and more efficiently**. + +### The Wedding Registry Anecdote + +I had the privilege of being "born" into eXtreme Programming. My first job _(back in 2007)_ was with a team deeply committed to XP values and practices. As the least experienced developer on the team, Test-Driven Development _(TDD)_ saved me _(among other practices)_. + +Thirteen years later, having always relied on testing and TDD, **I had to challenge my well-established assumptions**. What if we didn't need as much testing? Could there be situations where testing wasn't necessary at all? We don't need tests for a one-shot simple script or some [Spikes](../../02-glossary.md#spike), but we need tests for ambitious products, right? But where is the line? + +**I had to try driving without a seat belt**. After all, is a seat belt really necessary in my garage? What about in a parking lot? + +I just needed to find the right playground for my experiment. 🤔 + +The opportunity came when I was building a registry for my own wedding. Since I wasn't planning to marry twice, I assumed the code would be thrown away after the wedding anyway. So, **I decided to build the registry without tests and see what would happen**. + +Like someone unfamiliar with testing who can't resist manual testing, I found it difficult to resist the urge to write tests. Despite a few frustrating moments where I wished for a faster feedback loop than manual testing could provide, the development went smoothly. + +Why did it go well? Probably because I was working alone. **The application only took a few days to implement, so I was able to keep the entire architecture and all the implementation details in my head.** + +A year later, some friends asked if I could provide a similar registry for their wedding. I thought, "What a perfect opportunity for my experiment! **Let's turn this registry into a reusable one.**" + +That's when the trouble kicked in. Even though the codebase was simple, every change was more expensive than necessary. **I couldn't move as quickly as I wanted.** Without TDD, I had to overthink every modification. What should I test if I update this third-party dependency or that feature? Manually testing everything was exhausting, and the uncertainty of not knowing when I was done was frustrating. You think you're finished, and then you discover you missed something. + +In the end, here is what I learned from one of the least ambitious products I ever built or participated in building: + +- You might not need tests for least ambitious features, +- but beyond ambition and longevity, **most features will always be faster to develop with tests** _(given a good testing strategy)_. +- **Feature ambition is unstable** and can evolve in unexpected ways. +- **High cognitive capacity breeds overconfidence**: This experience also made me realize that testing helps me consume my cognitive capacity more wisely. In other words, just because you can keep the whole architecture in your head doesn't mean you should rely on it. The details will eventually fade away. + + + You never know what the journey will be, so you better put on your seat belt + before you start the car. + + +## How to Evaluate your Testing Strategy? + +Here are some metrics that can help you evaluate your testing strategy: + +- **🐞 Number of Regressions** +- **😰 Team Confidence** +- **♻️ Feedback Loop Speed** +- **🚨 Number of [False Positives](../../02-glossary.md#false-positive)** +- **🥷 Number of [False Negatives](../../02-glossary.md#false-negative)** + +### What About Code Coverage? + +As mentioned in my blog post about [Angular Template Code Coverage](https://marmicode.io/blog/angular-template-code-coverage-and-future-proof-testing): + +> While code coverage is a useful tool, it should be used wisely. Keep it as an indicator, not a rigid goal. +> +> > _Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes._ +> > +> > _-- Charles Goodhart_ +> +> In other words, **when a measure becomes a target, it ceases to be a good measure**. + +### Test Rating + +The idea of Test Rating is to rate each test based on the following criteria by incrementing a counter for each of them: + +- **False Positives**: if the test wrongly reports a bug that does not exist, +- **False Negatives**: if the test missed a bug it was supposed to catch, +- **Hero**: if the test saved the day by catching a bug that wouldn't have been caught before reaching other safety nets. + +These ratings are qualitative metrics that can help you learn which kinds of tests are providing value and which ones are not. You can then refine your testing strategy accordingly. + +```ts +/** + * @falsepositives 2 + * @falsenegatives 3 + * @hero 2 + */ +it('...', () => ...); +``` + +:::note +The Test Rating approach is an experimental idea that I am still refining. If you have any feedback or suggestions, feel free to reach out to me. +::: + +## Key Takeaways + +- **🚀 Prioritize Confidence, Cost, and Feedback Speed** over anything dogmatic. +- **🍦 Avoid the Ice Cream Cone Model**: do not over-rely on manual or Wide tests. +- **🍯 Adopt the Honeycomb Model**: focus on the widest Narrow tests. +- **🔍 Narrow Down Tests When Necessary**: when the System Under Test becomes too complex or widely shared. +- **🛡️ Strengthen Early Safety Nets**: add Narrow tests to areas where recurring bugs are caught by wide tests. +- **🧠 Design Tests with Future-Maintainers in Mind**: think about what will make future contributions smoother and more confident. +- **📊 Measure Testing Effectiveness using Qualitative Metrics**. diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/progressive-confidence.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/progressive-confidence.png new file mode 100644 index 0000000..c4aca24 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/progressive-confidence.png differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/safety-nets.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/safety-nets.png new file mode 100644 index 0000000..f0a4973 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/safety-nets.png differ diff --git a/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/test-pyramid.png b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/test-pyramid.png new file mode 100644 index 0000000..7f00631 Binary files /dev/null and b/apps/cookbook/docs/angular/01-testing/02-designing-a-pragmatic-testing-strategy/test-pyramid.png differ diff --git a/apps/cookbook/docs/angular/testing/02-vitest/01-why-vitest.md b/apps/cookbook/docs/angular/01-testing/03-vitest/01-why-vitest.md similarity index 100% rename from apps/cookbook/docs/angular/testing/02-vitest/01-why-vitest.md rename to apps/cookbook/docs/angular/01-testing/03-vitest/01-why-vitest.md diff --git a/apps/cookbook/docs/angular/testing/02-vitest/_category_.yml b/apps/cookbook/docs/angular/01-testing/03-vitest/_category_.yml similarity index 100% rename from apps/cookbook/docs/angular/testing/02-vitest/_category_.yml rename to apps/cookbook/docs/angular/01-testing/03-vitest/_category_.yml diff --git a/apps/cookbook/docs/angular/testing/02-vitest/vitest-module-graph.png b/apps/cookbook/docs/angular/01-testing/03-vitest/vitest-module-graph.png similarity index 100% rename from apps/cookbook/docs/angular/testing/02-vitest/vitest-module-graph.png rename to apps/cookbook/docs/angular/01-testing/03-vitest/vitest-module-graph.png diff --git a/apps/cookbook/docs/angular/testing/_category_.yml b/apps/cookbook/docs/angular/01-testing/_category_.yml similarity index 100% rename from apps/cookbook/docs/angular/testing/_category_.yml rename to apps/cookbook/docs/angular/01-testing/_category_.yml diff --git a/apps/cookbook/docs/angular/02-glossary.md b/apps/cookbook/docs/angular/02-glossary.md new file mode 100644 index 0000000..1076d9b --- /dev/null +++ b/apps/cookbook/docs/angular/02-glossary.md @@ -0,0 +1,142 @@ +--- +title: Glossary +slug: /angular/glossary +--- + +## 3X + +Explore, Expand, and Extract are the three phases of software development that Kent Beck extracted from his time working at Facebook. _Cf. [The Product Development Triathlon](https://medium.com/@kentbeck_7670/the-product-development-triathlon-6464e2763c46) and [3X Explore, Expand, Extract • Kent Beck • YOW! 2019](https://www.youtube.com/watch?v=lOcXdXRxFgA)_ + +## Affected Tests + +Affected tests are tests that are impacted by a change in the codebase. By running only the affected tests, developers can get faster feedback on the changes they have made. This can be achieved using features like [Nx Affected Graph](https://nx.dev/ci/features/affected#run-only-tasks-affected-by-a-pr) and/or [Vitest's `changed` option](https://vitest.dev/guide/cli.html#changed). + +## Canary Release + +Canary release is a technique used to reduce the risk of introducing a new feature or change to a large audience. By releasing the change to a small subset of users first, the team can monitor the impact and gather feedback before rolling out the change to the entire user base. + +## Cognitive Load + +Cognitive load refers to the amount of mental effort required to complete a task. In the context of testing, tests with high cognitive load can be difficult to understand and maintain, leading to decreased productivity and increased risk of errors. + +## Design Doc + +A design document outlines the design of a feature or system. It typically includes information about the problem being solved, the non-goals, the proposed solution, the alternative solutions, the trade-offs involved, and any other relevant details. Design docs can help teams align before diving into the implementation. + +## Dogfooding + +Dogfooding is the practice of using your own product or service internally before releasing it to the public. _([Rumor](https://www.nytimes.com/2022/11/14/business/dogfooding.html) has it that an executive of Whiskas would eat their dog food during shareholder meetings to prove its quality.)_ + +## eXtreme Programming _(XP)_ + +eXtreme Programming is a lightweight software development methodology that hasn't sold its soul to the devil of certification-driven business models. + +## False Negative + +A false negative is a test that did not detect a bug or issue that it should have detected. + +## False Positive + +A false positive is a test that reported a bug or issue that does not exist. + +## Humans + +Creative creatures that are bad at repetitive tasks like regression testing. + +## Isolation Modes + +Isolation modes refer to the different ways in which test files can be isolated from each other during execution. Vitest offers multiple isolation modes, including VM, threads, forks, and no isolate, allowing developers to choose the best trade-off between isolation and performance. + +## Narrow Tests + +Narrow tests are tests that are fast, easy to isolate and parallelize, and have a low cognitive load. They provide quick feedback and are designed to be run frequently during development. + +Cf. [Narrow Tests Definition](./01-testing/01-beyond-unit-vs-integration/index.mdx#narrow-tests) + +## Over-Specification + +Over-specification occurs when tests are too tightly coupled to the implementation details of the System Under Test. This can make tests brittle and hard to maintain, as any changes to the implementation will require corresponding changes to the tests. + +## Precise Tests + +Precise tests are tests that are specific and focused on a single aspect of the System Under Test. They should be easy to understand and provide clear feedback when they fail. + +_Cf. [Test Desiderata's Specific](#specific)_ + +## Symmetric to Production + +Symmetric to production refers to the similarity between the test environment and the production environment. By making the test environment as close to production as possible, developers can increase the likelihood that the tests will catch issues before they reach users. + +## System Under Test _(SUT)_ + +The System Under Test is the code or system that is being tested. It is the part of the codebase that is the focus of the test. + +## Test Desiderata + +Originally defined by Kent Beck, [Test Desiderata](https://testdesiderata.com/) is a set of properties we wish for our tests to have. The challenge is that some of these properties are incompatible with each other by nature, which means that we have to make trade-offs when designing our tests. + +#### Isolated + +Tests should return the same results regardless of the order in which they are run. + +#### Composable + +I should be able to test different dimensions of variability separately and combine the results. + +#### Deterministic + +If nothing changes, the test result shouldn’t change. + +#### Fast + +Tests should run quickly. + +#### Writeable + +Tests should be cheap to write relative to the cost of the code being tested. + +#### Readable + +Tests should be comprehensible for the reader, invoking the motivation for writing this particular test. + +#### Behavioral + +Tests should be sensitive to changes in the behavior of the code under test. If the behavior changes, the test result should change. + +#### Structure-Insensitive + +Tests should not change their result if the structure of the code changes. + +#### Automated + +Tests should run without human intervention. + +#### Specific + +If a test fails, the cause of the failure should be obvious. + +#### Predictive + +If the tests all pass, then the code under test should be suitable for production. + +#### Inspiring + +Passing the tests should inspire confidence. + +## Test-Driven Development _(TDD)_ + +Test-Driven Development _(TDD)_ is a software development process that relies on the repetition of a very short 3-phase development cycle: first, the developer writes an _(initially failing)_ test that defines a desired behavior, then produces the minimum amount of code that makes that test pass, and finally tidies up the new code to acceptable standards. + +## Test Doubles + +Test doubles are implementations used in place of real dependencies in tests. They can be used to simulate the behavior of real implementations and isolate the System Under Test from its dependencies. There are several types of test doubles, including dummies, stubs, spies, mocks, and fakes. + +## Vitest + +Vitest is a modern testing framework for JavaScript that is designed to be fast and flexible. It supports ECMAScript modules (ESM) out of the box and offers multiple isolation modes. Vitest provides a rich API for writing tests and is gaining traction in the Angular community as an alternative to Jest & Karma. + +## Wide Tests + +Wide tests are tests that are slower, harder to isolate, or have a higher cognitive load compared to Narrow tests. While sacrificing some of the properties of Narrow tests, Wide tests can provide a higher level of confidence by being more production-symmetric. They are useful for catching issues that narrow tests might miss. + +Cf. [Wide Tests Definition](./01-testing/01-beyond-unit-vs-integration/index.mdx#wide-tests) diff --git a/apps/cookbook/docs/nx/06-glossary.md b/apps/cookbook/docs/nx/06-glossary.md index 9fa4464..884cdab 100644 --- a/apps/cookbook/docs/nx/06-glossary.md +++ b/apps/cookbook/docs/nx/06-glossary.md @@ -1,9 +1,8 @@ --- +title: Glossary slug: /nx/glossary --- -# Glossary - ## Boundaries Boundaries define conceptual limits of a certain responsibility or concern within a software application. They help to establish clear separation between different parts of the system and promote modularity and maintainability. diff --git a/apps/cookbook/src/components/image-container.module.css b/apps/cookbook/src/components/image-container.module.css new file mode 100644 index 0000000..90a20bc --- /dev/null +++ b/apps/cookbook/src/components/image-container.module.css @@ -0,0 +1,11 @@ +.container img { + object-fit: contain; +} + +.small img { + height: 350px; +} + +.medium img { + height: 500px; +} diff --git a/apps/cookbook/src/components/image-container.tsx b/apps/cookbook/src/components/image-container.tsx new file mode 100644 index 0000000..13d0734 --- /dev/null +++ b/apps/cookbook/src/components/image-container.tsx @@ -0,0 +1,26 @@ +import { ReactNode } from 'react'; +import styles from './image-container.module.css'; +import clsx from 'clsx'; + +export function ImageContainer({ + children, + size, +}: { + children: ReactNode; + size: Size; +}) { + return ( +
{children}
+ ); +} + +export type Size = 'small' | 'medium'; + +function getClassName(size: Size): string | undefined { + switch (size) { + case 'small': + return styles.small; + case 'medium': + return styles.medium; + } +} diff --git a/apps/cookbook/src/css/custom.css b/apps/cookbook/src/css/custom.css index e4f3030..70c21e4 100644 --- a/apps/cookbook/src/css/custom.css +++ b/apps/cookbook/src/css/custom.css @@ -4,6 +4,8 @@ * work well for content-centric websites. */ +@import './mermaid.css'; + /* You can override the default Infima variables here. */ :root { --marmicode-primary: #380030; @@ -35,60 +37,3 @@ table { display: table; width: 100%; } - -.docusaurus-mermaid-container { - text-align: center; -} - -.docusaurus-mermaid-container svg .cluster rect { - fill: #fff4fd !important; - stroke: var(--marmicode-primary) !important; -} - -.docusaurus-mermaid-container svg .cluster .cluster-label span { - color: var(--marmicode-primary) !important; -} - -.docusaurus-mermaid-container svg .node rect { - fill: var(--marmicode-primary) !important; - stroke: white !important; -} - -.docusaurus-mermaid-container svg .node .label span { - color: white !important; -} - -[data-theme='light'] .docusaurus-mermaid-container svg .flowchart-link { - stroke: var(--marmicode-primary) !important; -} - -[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { - color: white !important; -} - -[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { - color: white !important; -} - -[data-theme='dark'] .docusaurus-mermaid-container svg .cluster rect { - fill: var(--marmicode-primary) !important; - stroke: white !important; -} - -[data-theme='dark'] - .docusaurus-mermaid-container - svg - .cluster - .cluster-label - span { - color: white !important; -} - -[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { - fill: #fff4fd !important; - stroke: white !important; -} - -[data-theme='dark'] .docusaurus-mermaid-container svg .node .label span { - color: var(--marmicode-primary) !important; -} diff --git a/apps/cookbook/src/css/mermaid.css b/apps/cookbook/src/css/mermaid.css new file mode 100644 index 0000000..0fe598a --- /dev/null +++ b/apps/cookbook/src/css/mermaid.css @@ -0,0 +1,56 @@ +.docusaurus-mermaid-container { + text-align: center; +} + +.docusaurus-mermaid-container svg .cluster rect { + fill: #fff4fd !important; + stroke: var(--marmicode-primary) !important; +} + +.docusaurus-mermaid-container svg .cluster .cluster-label span { + color: var(--marmicode-primary) !important; +} + +.docusaurus-mermaid-container svg .node rect { + fill: var(--marmicode-primary) !important; + stroke: white !important; +} + +.docusaurus-mermaid-container svg .node .label span { + color: white !important; +} + +[data-theme='light'] .docusaurus-mermaid-container svg .flowchart-link { + stroke: var(--marmicode-primary) !important; +} + +[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { + color: white !important; +} + +[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { + color: white !important; +} + +[data-theme='dark'] .docusaurus-mermaid-container svg .cluster rect { + fill: var(--marmicode-primary) !important; + stroke: white !important; +} + +[data-theme='dark'] + .docusaurus-mermaid-container + svg + .cluster + .cluster-label + span { + color: white !important; +} + +[data-theme='dark'] .docusaurus-mermaid-container svg .node rect { + fill: #fff4fd !important; + stroke: white !important; +} + +[data-theme='dark'] .docusaurus-mermaid-container svg .node .label span { + color: var(--marmicode-primary) !important; +}