Document best practices for writing tests for nodes and pipelines (#3782

)

* Update kedro-catalog-0.19.json (#3724)

* Update kedro-catalog-0.19.json

Signed-off-by: Anthony De Bortoli <[email protected]>

* Update set_up_vscode.md

Signed-off-by: Anthony De Bortoli <[email protected]>

---------

Signed-off-by: Anthony De Bortoli <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update project tests directory structure in docs

Signed-off-by: Ahdra Merali <[email protected]>

* Add docs on writing tests

Signed-off-by: Ahdra Merali <[email protected]>

* Drop dependency on toposort in favour of built-in graphlib (#3728)

* Replace toposort with graphlib (built-in from Python 3.9)

Signed-off-by: Ivan Danov <[email protected]>

* Create toposort groups only when needed

Signed-off-by: Ivan Danov <[email protected]>

* Update RELEASE.md and graphlib version constraints

Signed-off-by: Ivan Danov <[email protected]>

* Remove mypy-toposort

Signed-off-by: Ivan Danov <[email protected]>

* Ensure that the suggest resume test has no node ordering requirement

Signed-off-by: Ivan Danov <[email protected]>

* Ensure stable toposorting by grouping and ungrouping the result

Signed-off-by: Ivan Danov <[email protected]>

---------

Signed-off-by: Ivan Danov <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Optimise pipeline addition and creation (#3730)

* Create toposort groups only when needed
* Ensure that the suggest resume test has no node ordering requirement
* Ensure stable toposorting by grouping and ungrouping the result
* Delay toposorting until pipeline.nodes is used
* Avoid using .nodes when topological order or new copy is unneeded
* Copy the nodes only if tags are provided
* Remove unnecessary condition in self.nodes

Signed-off-by: Ivan Danov <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Expand robots.txt for Kedro-Viz and Kedro-Datasets docs (#3729)

* Add project to robots.txt

Signed-off-by: Dmitry Sorokin <[email protected]>

* Add EOF

Signed-off-by: Dmitry Sorokin <[email protected]>

---------

Signed-off-by: Dmitry Sorokin <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Kedro need more uv (#3740)

* Kedro need more uv

Signed-off-by: Nok <[email protected]>

* remove docker

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Nok <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Resolve all path in Kedro (#3742)

* Kedro need more uv

Signed-off-by: Nok <[email protected]>

* remove docker

Signed-off-by: Nok <[email protected]>

* fix broken type hint and resolve project path

Signed-off-by: Nok Lam Chan <[email protected]>

* fix type hint

Signed-off-by: Nok Lam Chan <[email protected]>

* remove duplicate logic

Signed-off-by: Nok Lam Chan <[email protected]>

* adding nok.py is definitely an accident

Signed-off-by: Nok Lam Chan <[email protected]>

* fix test

Signed-off-by: Nok Lam Chan <[email protected]>

* remove print

Signed-off-by: Nok Lam Chan <[email protected]>

* add test

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Nok <[email protected]>
Signed-off-by: Nok Lam Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Remove settings of rate limits and retries (#3769)

* double linkcheck limits

Signed-off-by: Nok Lam Chan <[email protected]>

* fix ratelimit

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Nok Lam Chan <[email protected]>
Signed-off-by: Nok <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Improve resume suggestions (#3719)

* Improve suggestions to resume a failed pipeline

- if dataset (or param) is persistent & shared, don't keep looking for ancestors
- only look for ancestors producing impersistent inputs
- minimize number of suggested nodes (= shorter message for the same pipeline)
- testable logic, tests cases outside of scenarios for sequential runner

- Use _EPHEMERAL attribute
- Move tests to separate file
- Docstring updates

---------

Signed-off-by: Ondrej Zacha <[email protected]>
Co-authored-by: Nok Lam Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Build docs fix (#3773)

* Ignored forbidden url

Signed-off-by: Elena Khaustova <[email protected]>

* Returned linkscheck retries

Signed-off-by: Elena Khaustova <[email protected]>

* Removed odd comment

Signed-off-by: Elena Khaustova <[email protected]>

---------

Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Clarify docs around custom resolvers (#3759)

* Updated custom resolver docs section

Signed-off-by: Elena Khaustova <[email protected]>

* Updated advanced configuration section for consistency

Signed-off-by: Elena Khaustova <[email protected]>

* Updated RELEASE.md

Signed-off-by: Elena Khaustova <[email protected]>

* Updated RELEASE.md

Signed-off-by: Elena Khaustova <[email protected]>

* Test linkcheck_workers decrease

Signed-off-by: Elena Khaustova <[email protected]>

* Increased the By default, the linkcheck_rate_limit_timeout to default

Signed-off-by: Elena Khaustova <[email protected]>

* Returned old docs build settings

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed typo

Signed-off-by: Elena Khaustova <[email protected]>

* Ignore forbidden url

Signed-off-by: Elena Khaustova <[email protected]>

* Returned linkcheck retries

Signed-off-by: Elena Khaustova <[email protected]>

---------

Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add mlruns to gitignore to avoid pushing  mlflow local runs to github (#3765)

* Add mlruns to gitignore to avoid pushing  mlflow local runs to github

Signed-off-by: Yolan Honoré-Rougé <[email protected]>

* update release.md

Signed-off-by: Yolan Honoré-Rougé <[email protected]>

---------

Signed-off-by: Yolan Honoré-Rougé <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: Nok Lam Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update the dependencies page in the docs (#3772)

* Update the dependencies page

Signed-off-by: Ankita Katiyar <[email protected]>

* Update docs/source/kedro_project_setup/dependencies.md

Signed-off-by: Ankita Katiyar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>

* Fix lint

Signed-off-by: Ankita Katiyar <[email protected]>

* Move the last line to notes

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Change pipeline test location to project root/tests (#3731)

* Change pipeline test location to project root/tests

Signed-off-by: lrcouto <[email protected]>

* Fix some test_pipeline tests

Signed-off-by: lrcouto <[email protected]>

* Change delete pipeline to account for new structure

Signed-off-by: lrcouto <[email protected]>

* Fix some tests

Signed-off-by: lrcouto <[email protected]>

* Change tests path on micropkg

Signed-off-by: lrcouto <[email protected]>

* Fix remaining tests

Signed-off-by: lrcouto <[email protected]>

* Add changes to release notes

Signed-off-by: lrcouto <[email protected]>

* Update file structure on micropackaging doc page

Signed-off-by: lrcouto <[email protected]>

---------

Signed-off-by: lrcouto <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add an option for kedro new to skip telemetry (#3701)

* First draft for telemetry consent flag on kedro new

Signed-off-by: lrcouto <[email protected]>

* Add functioning --telemetry option to kedro new

Signed-off-by: lrcouto <[email protected]>

* Update tests to acknowledge new flag

Signed-off-by: lrcouto <[email protected]>

* Add tests for kedro new --telemetry flag

Signed-off-by: lrcouto <[email protected]>

* Add changes to documentation and release notes

Signed-off-by: lrcouto <[email protected]>

* Minor change to docs

Signed-off-by: lrcouto <[email protected]>

* Lint

Signed-off-by: lrcouto <[email protected]>

* Remove outdated comment and correct type hint

Signed-off-by: lrcouto <[email protected]>

* Update docs/source/get_started/new_project.md

Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>

* Lint

Signed-off-by: lrcouto <[email protected]>

* Minor change on release note

Signed-off-by: lrcouto <[email protected]>

---------

Signed-off-by: lrcouto <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update documentation for OmegaConfigLoader (#3778)

* Update documentation for OmegaConfigLoader

Signed-off-by: Puneet Saini <[email protected]>

* Update RELEASE.md

Signed-off-by: Puneet Saini <[email protected]>

* Update RELEASE.md

Signed-off-by: Puneet Saini <[email protected]>

* Update ignore-names.txt

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Puneet Saini <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Fix path

Signed-off-by: Ahdra Merali <[email protected]>

* Lint

Signed-off-by: Ahdra Merali <[email protected]>

* Add changes to RELEASE.md

Signed-off-by: Ahdra Merali <[email protected]>

* Address comments from code review

Signed-off-by: Ahdra Merali <[email protected]>

* Empty

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Remove unneeded imports

Signed-off-by: Ahdra Merali <[email protected]>

* Change recommendation from pytest config to editable install

Signed-off-by: Ahdra Merali <[email protected]>

* Add negative testing example

Signed-off-by: Ahdra Merali <[email protected]>

* Replace Dict with dict

Signed-off-by: Ahdra Merali <[email protected]>

* Remove test classes

Signed-off-by: Ahdra Merali <[email protected]>

* Change the assert step for the integration test

Signed-off-by: Ahdra Merali <[email protected]>

* Fix error handling for OmegaConfigLoader (#3784)

* Update omegaconf_config.py

Signed-off-by: Puneet Saini <[email protected]>

* Update RELEASE.md

Signed-off-by: Puneet Saini <[email protected]>

* add a more complicated test case

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Puneet Saini <[email protected]>
Signed-off-by: Nok <[email protected]>
Co-authored-by: Nok <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add Simon Brugman to TSC (#3780)

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update technical_steering_committee.md (#3796)

Signed-off-by: Marcin Zabłocki <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Remove jmespath dependency (#3797)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update spaceflights tutorial and starter requirements for kedro-datasets optional dependencies (#3664)

* Update spaceflights tutorial and starter requirements

Signed-off-by: lrcouto <[email protected]>

* fix e2e tests

Signed-off-by: lrcouto <[email protected]>

* Fix e2e tests by distinguishing `kedro-datasets` dependency for different python versions (#3802)

Signed-off-by: Merel Theisen <[email protected]>

* Update docs/source/tutorial/tutorial_template.md

Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>

---------

Signed-off-by: lrcouto <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Consider Vale's suggestions

Signed-off-by: Ahdra Merali <[email protected]>

* Hide test in details

Signed-off-by: Ahdra Merali <[email protected]>

* Quick fix

Signed-off-by: Ahdra Merali <[email protected]>

* Typo (and wording changes)

Signed-off-by: Ahdra Merali <[email protected]>

* Update robots.txt (#3803)

Signed-off-by: Ankita Katiyar <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add a test for transcoding loops of 1 or more nodes (#3810)

Signed-off-by: Ivan Danov <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Ensure no nodes can depend on themselves even when transcoding is used (#3812)

* Factor out transcoding helpers into a private module

Signed-off-by: Ivan Danov <[email protected]>

* Ensure node input/output validation doesn't allow transcoded self-loops

Signed-off-by: Ivan Danov <[email protected]>

* Updated release note to avoid github warning

Signed-off-by: Elena Khaustova <[email protected]>

---------

Signed-off-by: Ivan Danov <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Co-authored-by: Elena Khaustova <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update UUID telemetry docs (#3805)

Signed-off-by: Dmitry Sorokin <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Change path to starters test (#3816)

Signed-off-by: lrcouto <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Move changes in RELEASE.md to docs section

Signed-off-by: Ahdra Merali <[email protected]>

* Change formatting

Signed-off-by: Ahdra Merali <[email protected]>

* Revert "Change formatting"

This reverts commit 9582a22.

Signed-off-by: Ahdra Merali <[email protected]>

* Apply changes from code review

Signed-off-by: Ahdra Merali <[email protected]>

* Add explanation on why cleanup isn't needed

Signed-off-by: Ahdra Merali <[email protected]>

* Change assert on successful pipeline to check logs

Signed-off-by: Ahdra Merali <[email protected]>

* Update description of integration test under pipeline slicing

Signed-off-by: Ahdra Merali <[email protected]>

* Missing formatting

Signed-off-by: Ahdra Merali <[email protected]>

* Update tests directory structure

Signed-off-by: Ahdra Merali <[email protected]>

---------

Signed-off-by: Anthony De Bortoli <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Signed-off-by: Ivan Danov <[email protected]>
Signed-off-by: Dmitry Sorokin <[email protected]>
Signed-off-by: Nok <[email protected]>
Signed-off-by: Nok Lam Chan <[email protected]>
Signed-off-by: Ondrej Zacha <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Yolan Honoré-Rougé <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: lrcouto <[email protected]>
Signed-off-by: L. R. Couto <[email protected]>
Signed-off-by: Puneet Saini <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Marcin Zabłocki <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Co-authored-by: Anthony De Bortoli <[email protected]>
Co-authored-by: Ivan Danov <[email protected]>
Co-authored-by: Dmitry Sorokin <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Nok Lam Chan <[email protected]>
Co-authored-by: Ondrej Zacha <[email protected]>
Co-authored-by: ElenaKhaustova <[email protected]>
Co-authored-by: Yolan Honoré-Rougé <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: L. R. Couto <[email protected]>
Co-authored-by: Puneet Saini <[email protected]>
Co-authored-by: Marcin Zabłocki <[email protected]>
Co-authored-by: Elena Khaustova <[email protected]>

RELEASE.md

@@ -6,7 +6,7 @@
 * Cookiecutter errors are shown in short format without the `--verbose` flag.
 * Kedro commands now work from any subdirectory within a Kedro project.
 * Kedro CLI now provides a better error message when project commands are run outside of a project i.e. `kedro run`
-* Adds the `--telemetry` flag to `kedro new`, allowing the user to register consent to have user analytics collected at the same time as the project is created.
+* Added the `--telemetry` flag to `kedro new`, allowing the user to register consent to have user analytics collected at the same time as the project is created.
 * Dropped the dependency on `toposort` in favour of the built-in `graphlib` module.
 * Improved the performance of `Pipeline` object creation and summing.
 * Improved suggestions to resume failed pipeline runs.
@@ -24,7 +24,8 @@
 * Methods `_is_project` and `_find_kedro_project` have been moved to `kedro.utils`. We recommend not using private methods in your code, but if you do, please update your code to use the new location.
 
 ## Documentation changes
-* Add missing description for `merge_strategy` argument in OmegaConfigLoader.
+* Added missing description for `merge_strategy` argument in OmegaConfigLoader.
+* Added documentation on best practices for testing nodes and pipelines.
 
 ## Community contributions
 Many thanks to the following Kedroids for contributing PRs to this release:

docs/source/development/automated_testing.md

@@ -19,6 +19,17 @@ There are many testing frameworks available for Python. One of the most popular
 
 Let's look at how you can start working with `pytest` in your Kedro project.
 
+### Prerequisite: Install your Kedro project
+
+Before getting started with `pytest`, it is important to ensure you have installed your project locally. This allows you to test different parts of your project by importing them into your test files.
+
+To install your project, navigate to your project root and run the following command:
+
+```bash
+pip install -e .
+```
+
+>**NOTE**: The option `-e` installs an editable version of your project, allowing you to make changes to the project files without needing to re-install them each time.
 ### Install `pytest`
 
 Install `pytest` as you would install other packages with `pip`, making sure your [project's virtual environment is active](../get_started/install.md#create-a-virtual-environment-for-your-kedro-project).
@@ -29,15 +40,15 @@ pip install pytest
 
 ### Create a `/tests` directory
 
-Now that `pytest` is installed, you will need a place to put your tests. Create a `/tests` folder in the `/src` directory of your project.
+Now that `pytest` is installed, you will need a place to put your tests. Create a `/tests` folder in the root directory of your project.
 
 ```bash
-mkdir <project_root>/src/tests
+mkdir <project_root>/tests
 ```
 
 ### Test directory structure
 
-The subdirectories in your project's `/tests` directory should mirror the directory structure of your project's `/src/<package_name>` directory. All files in the `/tests` folder should be named `test_<file_being_tested>.py`. See an example `/src` folder below.
+The subdirectories in your project's `/tests` directory should mirror the directory structure of your project's `/src/<package_name>` directory. All files in the `/tests` folder should be named `test_<file_being_tested>.py`. See an example `/tests` folder below.
 
 ```
 src
@@ -49,12 +60,12 @@ src
 │           │   nodes.py
 │           │   ...
 │
-└───tests
-│   └───pipelines
-│       └───dataprocessing
-│           │   ...
-│           │   test_nodes.py
-│           │   ...
+tests
+└───pipelines
+│   └───dataprocessing
+│       │   ...
+│       │   test_nodes.py
+│       │   ...
 ```
 
 ### Create an example test
@@ -96,6 +107,7 @@ Tests should be named as descriptively as possible, especially if you are workin
 
 You can read more about the [basics of using `pytest` on the getting started page](https://docs.pytest.org/en/7.1.x/getting-started.html). For help writing your own tests and using all of the features of `pytest`, see the [project documentation](https://docs.pytest.org/).
 
+
 ### Run your tests
 
 To run your tests, run `pytest` from within your project's root directory.
@@ -112,7 +124,7 @@ If you created the example test in the previous section, you should see the foll
 ...
 collected 1 item
 
-src/tests/test_run.py .                                                  [100%]
+tests/test_run.py .                                                  [100%]
 
 ============================== 1 passed in 0.38s ===============================
 ```

docs/source/tutorial/add_another_pipeline.md

@@ -29,15 +29,15 @@ First, take a look at the functions for the data science nodes in `src/spaceflig
 
 ```python
 import logging
-from typing import Dict, Tuple
+from typing import dict, Tuple
 
 import pandas as pd
 from sklearn.linear_model import LinearRegression
 from sklearn.metrics import r2_score
 from sklearn.model_selection import train_test_split
 
 
-def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
+def split_data(data: pd.DataFrame, parameters: dict[str, Any]) -> Tuple:
     """Splits data into features and targets training and test sets.
 
     Args:

docs/source/tutorial/spaceflights_tutorial.md

@@ -15,6 +15,7 @@ tutorial_template
 set_up_data
 create_a_pipeline
 add_another_pipeline
+test_a_project
 package_a_project
 spaceflights_tutorial_faqs
 ```