From 83eed17d2fcdbabf87ed72a888d011bc35856c44 Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Thu, 7 Dec 2023 18:28:46 +0800 Subject: [PATCH] Update documentations of Configuration default environments (#3321) * update docs Signed-off-by: Nok * add docs Signed-off-by: Nok * bubble up the common features and fix some level of headers Signed-off-by: Nok * update notebook with keywords Signed-off-by: Nok * remove default environments from notebook Signed-off-by: Nok * typo fix Signed-off-by: Nok * Revert "bubble up the common features and fix some level of headers" This reverts commit e6b95870a7676cc1df0e5c1e269e5624f6cc0bcf. * Fix the header Signed-off-by: Nok * add example for standalone config Signed-off-by: Nok * Apply suggestions from code review Co-authored-by: Jo Stichbury Signed-off-by: Nok Lam Chan * Update docs/source/configuration/configuration_basics.md Co-authored-by: Jo Stichbury Signed-off-by: Nok Lam Chan * fix grammar Signed-off-by: Nok * fix format Signed-off-by: Nok * fix format Signed-off-by: Nok * fix grammar Signed-off-by: Nok * missing plurals Signed-off-by: Nok --------- Signed-off-by: Nok Signed-off-by: Nok Lam Chan Co-authored-by: Jo Stichbury --- .../configuration/advanced_configuration.md | 63 ++++++++++++++++++- .../configuration/configuration_basics.md | 4 +- .../add_kedro_to_a_notebook.ipynb | 8 +-- .../add_kedro_to_a_notebook.md | 8 +-- 4 files changed, 72 insertions(+), 11 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 041f23cd6e..2ad41ce6f7 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -3,6 +3,7 @@ The documentation on [configuration](./configuration_basics.md) describes how to By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class. +## Advanced configuration for Kedro projects This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects: * [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) @@ -18,7 +19,7 @@ This page also contains a set of guidance for advanced configuration requirement * [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader) -## How to use a custom configuration loader +### How to use a custom configuration loader You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: ```python @@ -75,6 +76,10 @@ CONFIG_LOADER_ARGS = { ### How to bypass the configuration loading rules You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. +For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials). + +Alternatively, if you are using config loader as a standalone component, you can override configuration as follows: + ```{code-block} python :lineno-start: 10 :emphasize-lines: 8 @@ -310,3 +315,59 @@ CONFIG_LOADER_ARGS = { If no merge strategy is defined, the default destructive strategy will be applied. Note that this merge strategy setting only applies to configuration files in **different** environments. When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys. + + +## Advanced configuration without a full Kedro project + In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project has a `base` and `local` environment. +However, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). + +### Read configuration +The config loader can work without a Kedro project structure. +```bash +tree . +. +└── parameters.yml +``` + +```yaml +# parameters.yml +learning_rate: 0.01 +train_test_ratio: 0.7 +``` + +```python +from kedro.config import OmegaConfigLoader +config_loader = OmegaConfigLoader(conf_source=".") + +# Optionally, you can also use environments +# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local") + +>>> config_loader["parameters"] +{'learning_rate': 0.01, 'train_test_ratio': 0.7} +``` + +For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md) + +### How to use Custom Resolvers with `OmegaConfigLoader` +You can register custom resolvers to use non-primitive types for parameters. + +```yaml +# parameters.yml +polars_float64: "${polars: Float64}" +today: "${today:}" +``` + +```python +import polars as pl +from datetime import date + +from kedro.config import OmegaConfigLoader + +custom_resolvers = {"polars": lambda x: getattr(pl, x), + "today": lambda: date.today()} + +# Register custom resolvers +config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers) +>>> print(config_loader["parameters"]) +{'polars_float64': Float64, 'today': datetime.date(2023, 11, 23)} +``` diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index 188d8c9c17..e3df81e25e 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -44,7 +44,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con ## Configuration environments A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments. -By default, Kedro has a `base` and a `local` environment. +By default, Kedro projects have a `base` and a `local` environment. ### Base In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments. @@ -82,7 +82,7 @@ Configuration files will be matched according to file name and type rules. Suppo * *Either* of the following is true: * filename starts with `catalog` * file is located in a subfolder whose name is prefixed with `catalog` -* *And* file extension is one of the following: yaml`, `yml`, or `json` +* *And* file extension is one of the following: `yaml`, `yml`, or `json` ### Configuration patterns Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes. diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb index 4382cae12f..4d065122a9 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb @@ -436,7 +436,7 @@ "source": [ "from kedro.config import OmegaConfigLoader\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")" + "conf_loader = OmegaConfigLoader(conf_source=\".\")" ] }, { @@ -548,7 +548,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "\n", "# Create the DataCatalog instance from the configuration\n", @@ -584,7 +584,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "conf_params = conf_loader[\"parameters\"]\n", "\n", @@ -804,7 +804,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "conf_params = conf_loader[\"parameters\"]\n", "\n", diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md index 47f31c5272..0091ab83ad 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md @@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo ```python from kedro.config import OmegaConfigLoader -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") ``` ```python @@ -348,7 +348,7 @@ To load `catalog.yml` the code is as follows: from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] # Create the DataCatalog instance from the configuration @@ -372,7 +372,7 @@ Code in a Kedro project runs in one or more pipelines, where a pipeline is a ser from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] conf_params = conf_loader["parameters"] @@ -546,7 +546,7 @@ And that's it. The notebook code has been refactored into a series of functions. from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] conf_params = conf_loader["parameters"]