From a9c9c36c80668f6c43d970a3e381bb8c9121f5fd Mon Sep 17 00:00:00 2001 From: Nok Date: Mon, 20 Nov 2023 09:17:03 +0000 Subject: [PATCH 01/17] update docs Signed-off-by: Nok --- docs/source/configuration/configuration_basics.md | 2 +- .../notebook-example/add_kedro_to_a_notebook.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index 58ed579444..af9ba95859 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -22,7 +22,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con ## Configuration environments A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments. -By default, Kedro has a `base` and a `local` environment. +By default, Kedro Framework has a `base` and a `local` environment. ### Base In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments. diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md index 47f31c5272..e6fad7e5c8 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md @@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo ```python from kedro.config import OmegaConfigLoader -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(".") ``` ```python From c902418e1f0fdf4e98b76ca9e2e5b1284b70b538 Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 22 Nov 2023 11:13:00 +0000 Subject: [PATCH 02/17] add docs Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 94427b19b0..7cb0f15f0a 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -74,6 +74,10 @@ CONFIG_LOADER_ARGS = { ### How to bypass the configuration loading rules You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. +For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). + +Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: + ```{code-block} python :lineno-start: 10 :emphasize-lines: 8 From e6b95870a7676cc1df0e5c1e269e5624f6cc0bcf Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 22 Nov 2023 11:19:56 +0000 Subject: [PATCH 03/17] bubble up the common features and fix some level of headers Signed-off-by: Nok --- .../configuration/advanced_configuration.md | 248 +++++++++--------- 1 file changed, 131 insertions(+), 117 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 7cb0f15f0a..3b28193ab5 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -5,92 +5,74 @@ By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaCo This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects: -* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) -* [How to change which configuration files are loaded](#how-to-change-which-configuration-files-are-loaded) -* [How to ensure non default configuration files get loaded](#how-to-ensure-non-default-configuration-files-get-loaded) -* [How to bypass the configuration loading rules](#how-to-bypass-the-configuration-loading-rules) -* [How to do templating with the `OmegaConfigLoader`](#how-to-do-templating-with-the-omegaconfigloader) * [How to use global variables with the `OmegaConfigLoader`](#how-to-use-global-variables-with-the-omegaconfigloader) * [How to override configuration with runtime parameters with the `OmegaConfigLoader`](#how-to-override-configuration-with-runtime-parameters-with-the-omegaconfigloader) +* [How to do templating with the `OmegaConfigLoader`](#how-to-do-templating-with-the-omegaconfigloader) +* [How to change which configuration files are loaded](#how-to-change-which-configuration-files-are-loaded) +* [How to ensure non default configuration files get loaded](#how-to-ensure-non-default-configuration-files-get-loaded) * [How to use resolvers in the `OmegaConfigLoader`](#how-to-use-resolvers-in-the-omegaconfigloader) +* [How to bypass the configuration loading rules](#how-to-bypass-the-configuration-loading-rules) +* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) * [How to load credentials through environment variables with `OmegaConfigLoader`](#how-to-load-credentials-through-environment-variables) * [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader) -## How to use a custom configuration loader -You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: - -```python -from kedro.config import AbstractConfigLoader - - -class CustomConfigLoader(AbstractConfigLoader): - def __init__( - self, - conf_source: str, - env: str = None, - runtime_params: Dict[str, Any] = None, - ): - super().__init__( - conf_source=conf_source, env=env, runtime_params=runtime_params - ) +### How to use global variables with the `OmegaConfigLoader` +From Kedro `0.18.13`, you can use variable interpolation in your configurations using "globals" with `OmegaConfigLoader`. +The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types, such as catalog and parameters. +By default, these global variables are assumed to be in files called `globals.yml` in any of your environments. If you want to configure the naming patterns for the files that contain your global variables, +you can do so [by overwriting the `globals` key in `config_patterns`](#how-to-change-which-configuration-files-are-loaded). You can also [bypass the configuration loading](#how-to-bypass-the-configuration-loading-rules) +to directly set the global variables in `OmegaConfigLoader`. - # Custom implementation +Suppose you have global variables located in the file `conf/base/globals.yml`: +```yaml +my_global_value: 45 +dataset_type: + csv: pandas.CSVDataset ``` -To use this custom configuration loader, set it as the project configuration loader in `src//settings.py` as follows: -```python -from package_name.custom_configloader import CustomConfigLoader - -CONFIG_LOADER_CLASS = CustomConfigLoader +You can access these global variables in your catalog or parameters config files with a `globals` resolver like this: +`conf/base/parameters.yml`: +```yaml +my_param : "${globals:my_global_value}" ``` - - -### How to change which configuration files are loaded -If you want to change the patterns that the configuration loader uses to find the files to load you need to set the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). -For example, if your `parameters` files are using a `params` naming convention instead of `parameters` (e.g. `params.yml`) you need to update `CONFIG_LOADER_ARGS` as follows: - -```python -CONFIG_LOADER_ARGS = { - "config_patterns": { - "parameters": ["params*", "params*/**", "**/params*"], - } -} +`conf/base/catalog.yml`: +```yaml +companies: + filepath: data/01_raw/companies.csv + type: "${globals:dataset_type.csv}" ``` - -By changing this setting, the default behaviour for loading parameters will be replaced, while the other configuration patterns will remain in their default state. - -### How to ensure non default configuration files get loaded -You can add configuration patterns to match files other than `parameters`, `credentials`, and `catalog` by setting the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). -For example, if you want to load Spark configuration files you need to update `CONFIG_LOADER_ARGS` as follows: - -```python -CONFIG_LOADER_ARGS = { - "config_patterns": { - "spark": ["spark*/"], - } -} +You can also provide a default value to be used in case the global variable does not exist: +```yaml +my_param: "${globals: nonexistent_global, 23}" ``` +If there are duplicate keys in the globals files in your base and runtime environments, the values in the runtime environment +overwrite the values in your base environment. -### How to bypass the configuration loading rules -You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. - -For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). - -Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: - -```{code-block} python -:lineno-start: 10 -:emphasize-lines: 8 -from kedro.config import OmegaConfigLoader -from kedro.framework.project import settings +### How to override configuration with runtime parameters with the `OmegaConfigLoader` -conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = OmegaConfigLoader(conf_source=conf_path) +Kedro allows you to [specify runtime parameters for the `kedro run` command with the `--params` CLI option](parameters.md#how-to-specify-parameters-at-runtime). These runtime parameters +are added to the `KedroContext` and merged with parameters from the configuration files to be used in your project's pipelines and nodes. From Kedro `0.18.14`, you can use the +`runtime_params` resolver to indicate that you want to override values of certain keys in your configuration with runtime parameters provided through the CLI option. +This resolver can be used across different configuration types, such as parameters, catalog, and more, except for "globals". -# Bypass configuration patterns by setting the key and values directly on the config loader instance. -conf_loader["catalog"] = {"catalog_config": "something_new"} +Consider this `parameters.yml` file: +```yaml +model_options: + random_state: "${runtime_params:random}" +``` +This will allow you to pass a runtime parameter named `random` through the CLI to specify the value of `model_options.random_state` in your project's parameters: +```bash +kedro run --params random=3 ``` +You can also specify a default value to be used in case the runtime parameter is not specified with the `kedro run` command. Consider this catalog entry: +```yaml +companies: + type: pandas.CSVDataset + filepath: "${runtime_params:folder, 'data/01_raw'}/companies.csv" +``` +If the `folder` parameter is not passed through the CLI `--params` option with `kedro run`, the default value `'data/01_raw/'` is used for the `filepath`. + ### How to do templating with the `OmegaConfigLoader` #### Parameters @@ -137,60 +119,35 @@ Since both of the file names (`catalog.yml` and `catalog_globals.yml`) match the #### Other configuration files It's also possible to use variable interpolation in configuration files other than parameters and catalog, such as custom spark or mlflow configuration. This works in the same way as variable interpolation in parameter files. You can still use the underscore for the templated values if you want, but it's not mandatory like it is for catalog files. -### How to use global variables with the `OmegaConfigLoader` -From Kedro `0.18.13`, you can use variable interpolation in your configurations using "globals" with `OmegaConfigLoader`. -The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types, such as catalog and parameters. -By default, these global variables are assumed to be in files called `globals.yml` in any of your environments. If you want to configure the naming patterns for the files that contain your global variables, -you can do so [by overwriting the `globals` key in `config_patterns`](#how-to-change-which-configuration-files-are-loaded). You can also [bypass the configuration loading](#how-to-bypass-the-configuration-loading-rules) -to directly set the global variables in `OmegaConfigLoader`. -Suppose you have global variables located in the file `conf/base/globals.yml`: -```yaml -my_global_value: 45 -dataset_type: - csv: pandas.CSVDataset -``` -You can access these global variables in your catalog or parameters config files with a `globals` resolver like this: -`conf/base/parameters.yml`: -```yaml -my_param : "${globals:my_global_value}" -``` -`conf/base/catalog.yml`: -```yaml -companies: - filepath: data/01_raw/companies.csv - type: "${globals:dataset_type.csv}" -``` -You can also provide a default value to be used in case the global variable does not exist: -```yaml -my_param: "${globals: nonexistent_global, 23}" +### How to change which configuration files are loaded +If you want to change the patterns that the configuration loader uses to find the files to load you need to set the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). +For example, if your `parameters` files are using a `params` naming convention instead of `parameters` (e.g. `params.yml`) you need to update `CONFIG_LOADER_ARGS` as follows: + +```python +CONFIG_LOADER_ARGS = { + "config_patterns": { + "parameters": ["params*", "params*/**", "**/params*"], + } +} ``` -If there are duplicate keys in the globals files in your base and runtime environments, the values in the runtime environment -overwrite the values in your base environment. -### How to override configuration with runtime parameters with the `OmegaConfigLoader` +By changing this setting, the default behaviour for loading parameters will be replaced, while the other configuration patterns will remain in their default state. -Kedro allows you to [specify runtime parameters for the `kedro run` command with the `--params` CLI option](parameters.md#how-to-specify-parameters-at-runtime). These runtime parameters -are added to the `KedroContext` and merged with parameters from the configuration files to be used in your project's pipelines and nodes. From Kedro `0.18.14`, you can use the -`runtime_params` resolver to indicate that you want to override values of certain keys in your configuration with runtime parameters provided through the CLI option. -This resolver can be used across different configuration types, such as parameters, catalog, and more, except for "globals". -Consider this `parameters.yml` file: -```yaml -model_options: - random_state: "${runtime_params:random}" -``` -This will allow you to pass a runtime parameter named `random` through the CLI to specify the value of `model_options.random_state` in your project's parameters: -```bash -kedro run --params random=3 -``` -You can also specify a default value to be used in case the runtime parameter is not specified with the `kedro run` command. Consider this catalog entry: -```yaml -companies: - type: pandas.CSVDataset - filepath: "${runtime_params:folder, 'data/01_raw'}/companies.csv" +### How to ensure non default configuration files get loaded +You can add configuration patterns to match files other than `parameters`, `credentials`, and `catalog` by setting the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). +For example, if you want to load Spark configuration files you need to update `CONFIG_LOADER_ARGS` as follows: + +```python +CONFIG_LOADER_ARGS = { + "config_patterns": { + "spark": ["spark*/"], + } +} ``` -If the `folder` parameter is not passed through the CLI `--params` option with `kedro run`, the default value `'data/01_raw/'` is used for the `filepath`. + + ### How to use resolvers in the `OmegaConfigLoader` Instead of hard-coding values in your configuration files, you can also dynamically compute them using [`OmegaConf`'s @@ -254,6 +211,63 @@ CONFIG_LOADER_ARGS = { } } ``` + + + + +### How to bypass the configuration loading rules +You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. + +For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). + +Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: + +```{code-block} python +:lineno-start: 10 +:emphasize-lines: 8 + +from kedro.config import OmegaConfigLoader +from kedro.framework.project import settings + +conf_path = str(project_path / settings.CONF_SOURCE) +conf_loader = OmegaConfigLoader(conf_source=conf_path) + +# Bypass configuration patterns by setting the key and values directly on the config loader instance. +conf_loader["catalog"] = {"catalog_config": "something_new"} +``` + + + + +### How to use a custom configuration loader +You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: + +```python +from kedro.config import AbstractConfigLoader + + +class CustomConfigLoader(AbstractConfigLoader): + def __init__( + self, + conf_source: str, + env: str = None, + runtime_params: Dict[str, Any] = None, + ): + super().__init__( + conf_source=conf_source, env=env, runtime_params=runtime_params + ) + + # Custom implementation +``` +To use this custom configuration loader, set it as the project configuration loader in `src//settings.py` as follows: +```python +from package_name.custom_configloader import CustomConfigLoader + +CONFIG_LOADER_CLASS = CustomConfigLoader +``` + + + ### How to load credentials through environment variables The [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) enables you to load credentials from environment variables. To achieve this you have to use the `OmegaConfigLoader` and the `omegaconf` [`oc.env` resolver](https://omegaconf.readthedocs.io/en/2.3_branch/custom_resolvers.html#oc-env). You can use the `oc.env` resolver to access credentials from environment variables in your `credentials.yml`: From 0f78bb6221ebc095bac0ad04878b613bc08d9c33 Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 22 Nov 2023 11:25:26 +0000 Subject: [PATCH 04/17] update notebook with keywords Signed-off-by: Nok --- .../notebook-example/add_kedro_to_a_notebook.ipynb | 2 +- .../notebook-example/add_kedro_to_a_notebook.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb index 4382cae12f..91d17c9dcd 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb @@ -436,7 +436,7 @@ "source": [ "from kedro.config import OmegaConfigLoader\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")" + "conf_loader = OmegaConfigLoader(conf_source=\".\")" ] }, { diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md index e6fad7e5c8..ab10195bff 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md @@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo ```python from kedro.config import OmegaConfigLoader -conf_loader = OmegaConfigLoader(".") +conf_loader = OmegaConfigLoader(conf_source=".") ``` ```python From 11a3a1b423f8a19d202c0e0efdab5a40a51e10dd Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 22 Nov 2023 11:28:55 +0000 Subject: [PATCH 05/17] remove default environments from notebook Signed-off-by: Nok --- .../notebook-example/add_kedro_to_a_notebook.ipynb | 6 +++--- .../notebook-example/add_kedro_to_a_notebook.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb index 91d17c9dcd..4d065122a9 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb @@ -548,7 +548,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "\n", "# Create the DataCatalog instance from the configuration\n", @@ -584,7 +584,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "conf_params = conf_loader[\"parameters\"]\n", "\n", @@ -804,7 +804,7 @@ "from kedro.config import OmegaConfigLoader\n", "from kedro.io import DataCatalog\n", "\n", - "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n", + "conf_loader = OmegaConfigLoader(conf_source=\".\")\n", "conf_catalog = conf_loader[\"catalog\"]\n", "conf_params = conf_loader[\"parameters\"]\n", "\n", diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md index ab10195bff..0091ab83ad 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md @@ -348,7 +348,7 @@ To load `catalog.yml` the code is as follows: from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] # Create the DataCatalog instance from the configuration @@ -372,7 +372,7 @@ Code in a Kedro project runs in one or more pipelines, where a pipeline is a ser from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] conf_params = conf_loader["parameters"] @@ -546,7 +546,7 @@ And that's it. The notebook code has been refactored into a series of functions. from kedro.config import OmegaConfigLoader from kedro.io import DataCatalog -conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="") +conf_loader = OmegaConfigLoader(conf_source=".") conf_catalog = conf_loader["catalog"] conf_params = conf_loader["parameters"] From 0713ac3001ddbb1dff7d980e25fa3b29ad2c8986 Mon Sep 17 00:00:00 2001 From: Nok Date: Wed, 22 Nov 2023 11:29:46 +0000 Subject: [PATCH 06/17] typo fix Signed-off-by: Nok --- docs/source/configuration/configuration_basics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index af9ba95859..1851243017 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -60,7 +60,7 @@ Configuration files will be matched according to file name and type rules. Suppo * *Either* of the following is true: * filename starts with `catalog` * file is located in a subfolder whose name is prefixed with `catalog` -* *And* file extension is one of the following: yaml`, `yml`, or `json` +* *And* file extension is one of the following: `yaml`, `yml`, or `json` ### Configuration patterns Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes. From 65966b9d3f111372ebca0a87f0f2a08572af8c75 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 06:54:08 +0000 Subject: [PATCH 07/17] Revert "bubble up the common features and fix some level of headers" This reverts commit e6b95870a7676cc1df0e5c1e269e5624f6cc0bcf. --- .../configuration/advanced_configuration.md | 248 +++++++++--------- 1 file changed, 117 insertions(+), 131 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 3b28193ab5..7cb0f15f0a 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -5,74 +5,92 @@ By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaCo This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects: -* [How to use global variables with the `OmegaConfigLoader`](#how-to-use-global-variables-with-the-omegaconfigloader) -* [How to override configuration with runtime parameters with the `OmegaConfigLoader`](#how-to-override-configuration-with-runtime-parameters-with-the-omegaconfigloader) -* [How to do templating with the `OmegaConfigLoader`](#how-to-do-templating-with-the-omegaconfigloader) +* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) * [How to change which configuration files are loaded](#how-to-change-which-configuration-files-are-loaded) * [How to ensure non default configuration files get loaded](#how-to-ensure-non-default-configuration-files-get-loaded) -* [How to use resolvers in the `OmegaConfigLoader`](#how-to-use-resolvers-in-the-omegaconfigloader) * [How to bypass the configuration loading rules](#how-to-bypass-the-configuration-loading-rules) -* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) +* [How to do templating with the `OmegaConfigLoader`](#how-to-do-templating-with-the-omegaconfigloader) +* [How to use global variables with the `OmegaConfigLoader`](#how-to-use-global-variables-with-the-omegaconfigloader) +* [How to override configuration with runtime parameters with the `OmegaConfigLoader`](#how-to-override-configuration-with-runtime-parameters-with-the-omegaconfigloader) +* [How to use resolvers in the `OmegaConfigLoader`](#how-to-use-resolvers-in-the-omegaconfigloader) * [How to load credentials through environment variables with `OmegaConfigLoader`](#how-to-load-credentials-through-environment-variables) * [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader) -### How to use global variables with the `OmegaConfigLoader` -From Kedro `0.18.13`, you can use variable interpolation in your configurations using "globals" with `OmegaConfigLoader`. -The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types, such as catalog and parameters. -By default, these global variables are assumed to be in files called `globals.yml` in any of your environments. If you want to configure the naming patterns for the files that contain your global variables, -you can do so [by overwriting the `globals` key in `config_patterns`](#how-to-change-which-configuration-files-are-loaded). You can also [bypass the configuration loading](#how-to-bypass-the-configuration-loading-rules) -to directly set the global variables in `OmegaConfigLoader`. +## How to use a custom configuration loader +You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: -Suppose you have global variables located in the file `conf/base/globals.yml`: -```yaml -my_global_value: 45 -dataset_type: - csv: pandas.CSVDataset +```python +from kedro.config import AbstractConfigLoader + + +class CustomConfigLoader(AbstractConfigLoader): + def __init__( + self, + conf_source: str, + env: str = None, + runtime_params: Dict[str, Any] = None, + ): + super().__init__( + conf_source=conf_source, env=env, runtime_params=runtime_params + ) + + # Custom implementation ``` -You can access these global variables in your catalog or parameters config files with a `globals` resolver like this: -`conf/base/parameters.yml`: -```yaml -my_param : "${globals:my_global_value}" +To use this custom configuration loader, set it as the project configuration loader in `src//settings.py` as follows: +```python +from package_name.custom_configloader import CustomConfigLoader + +CONFIG_LOADER_CLASS = CustomConfigLoader ``` -`conf/base/catalog.yml`: -```yaml -companies: - filepath: data/01_raw/companies.csv - type: "${globals:dataset_type.csv}" + + +### How to change which configuration files are loaded +If you want to change the patterns that the configuration loader uses to find the files to load you need to set the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). +For example, if your `parameters` files are using a `params` naming convention instead of `parameters` (e.g. `params.yml`) you need to update `CONFIG_LOADER_ARGS` as follows: + +```python +CONFIG_LOADER_ARGS = { + "config_patterns": { + "parameters": ["params*", "params*/**", "**/params*"], + } +} ``` -You can also provide a default value to be used in case the global variable does not exist: -```yaml -my_param: "${globals: nonexistent_global, 23}" + +By changing this setting, the default behaviour for loading parameters will be replaced, while the other configuration patterns will remain in their default state. + +### How to ensure non default configuration files get loaded +You can add configuration patterns to match files other than `parameters`, `credentials`, and `catalog` by setting the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). +For example, if you want to load Spark configuration files you need to update `CONFIG_LOADER_ARGS` as follows: + +```python +CONFIG_LOADER_ARGS = { + "config_patterns": { + "spark": ["spark*/"], + } +} ``` -If there are duplicate keys in the globals files in your base and runtime environments, the values in the runtime environment -overwrite the values in your base environment. +### How to bypass the configuration loading rules +You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. -### How to override configuration with runtime parameters with the `OmegaConfigLoader` +For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). -Kedro allows you to [specify runtime parameters for the `kedro run` command with the `--params` CLI option](parameters.md#how-to-specify-parameters-at-runtime). These runtime parameters -are added to the `KedroContext` and merged with parameters from the configuration files to be used in your project's pipelines and nodes. From Kedro `0.18.14`, you can use the -`runtime_params` resolver to indicate that you want to override values of certain keys in your configuration with runtime parameters provided through the CLI option. -This resolver can be used across different configuration types, such as parameters, catalog, and more, except for "globals". +Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: -Consider this `parameters.yml` file: -```yaml -model_options: - random_state: "${runtime_params:random}" -``` -This will allow you to pass a runtime parameter named `random` through the CLI to specify the value of `model_options.random_state` in your project's parameters: -```bash -kedro run --params random=3 -``` -You can also specify a default value to be used in case the runtime parameter is not specified with the `kedro run` command. Consider this catalog entry: -```yaml -companies: - type: pandas.CSVDataset - filepath: "${runtime_params:folder, 'data/01_raw'}/companies.csv" -``` -If the `folder` parameter is not passed through the CLI `--params` option with `kedro run`, the default value `'data/01_raw/'` is used for the `filepath`. +```{code-block} python +:lineno-start: 10 +:emphasize-lines: 8 + +from kedro.config import OmegaConfigLoader +from kedro.framework.project import settings +conf_path = str(project_path / settings.CONF_SOURCE) +conf_loader = OmegaConfigLoader(conf_source=conf_path) + +# Bypass configuration patterns by setting the key and values directly on the config loader instance. +conf_loader["catalog"] = {"catalog_config": "something_new"} +``` ### How to do templating with the `OmegaConfigLoader` #### Parameters @@ -119,35 +137,60 @@ Since both of the file names (`catalog.yml` and `catalog_globals.yml`) match the #### Other configuration files It's also possible to use variable interpolation in configuration files other than parameters and catalog, such as custom spark or mlflow configuration. This works in the same way as variable interpolation in parameter files. You can still use the underscore for the templated values if you want, but it's not mandatory like it is for catalog files. +### How to use global variables with the `OmegaConfigLoader` +From Kedro `0.18.13`, you can use variable interpolation in your configurations using "globals" with `OmegaConfigLoader`. +The benefit of using globals over regular variable interpolation is that the global variables are shared across different configuration types, such as catalog and parameters. +By default, these global variables are assumed to be in files called `globals.yml` in any of your environments. If you want to configure the naming patterns for the files that contain your global variables, +you can do so [by overwriting the `globals` key in `config_patterns`](#how-to-change-which-configuration-files-are-loaded). You can also [bypass the configuration loading](#how-to-bypass-the-configuration-loading-rules) +to directly set the global variables in `OmegaConfigLoader`. -### How to change which configuration files are loaded -If you want to change the patterns that the configuration loader uses to find the files to load you need to set the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). -For example, if your `parameters` files are using a `params` naming convention instead of `parameters` (e.g. `params.yml`) you need to update `CONFIG_LOADER_ARGS` as follows: - -```python -CONFIG_LOADER_ARGS = { - "config_patterns": { - "parameters": ["params*", "params*/**", "**/params*"], - } -} +Suppose you have global variables located in the file `conf/base/globals.yml`: +```yaml +my_global_value: 45 +dataset_type: + csv: pandas.CSVDataset ``` +You can access these global variables in your catalog or parameters config files with a `globals` resolver like this: +`conf/base/parameters.yml`: +```yaml +my_param : "${globals:my_global_value}" +``` +`conf/base/catalog.yml`: +```yaml +companies: + filepath: data/01_raw/companies.csv + type: "${globals:dataset_type.csv}" +``` +You can also provide a default value to be used in case the global variable does not exist: +```yaml +my_param: "${globals: nonexistent_global, 23}" +``` +If there are duplicate keys in the globals files in your base and runtime environments, the values in the runtime environment +overwrite the values in your base environment. -By changing this setting, the default behaviour for loading parameters will be replaced, while the other configuration patterns will remain in their default state. - +### How to override configuration with runtime parameters with the `OmegaConfigLoader` -### How to ensure non default configuration files get loaded -You can add configuration patterns to match files other than `parameters`, `credentials`, and `catalog` by setting the `CONFIG_LOADER_ARGS` variable in [`src//settings.py`](../kedro_project_setup/settings.md). -For example, if you want to load Spark configuration files you need to update `CONFIG_LOADER_ARGS` as follows: +Kedro allows you to [specify runtime parameters for the `kedro run` command with the `--params` CLI option](parameters.md#how-to-specify-parameters-at-runtime). These runtime parameters +are added to the `KedroContext` and merged with parameters from the configuration files to be used in your project's pipelines and nodes. From Kedro `0.18.14`, you can use the +`runtime_params` resolver to indicate that you want to override values of certain keys in your configuration with runtime parameters provided through the CLI option. +This resolver can be used across different configuration types, such as parameters, catalog, and more, except for "globals". -```python -CONFIG_LOADER_ARGS = { - "config_patterns": { - "spark": ["spark*/"], - } -} +Consider this `parameters.yml` file: +```yaml +model_options: + random_state: "${runtime_params:random}" ``` - - +This will allow you to pass a runtime parameter named `random` through the CLI to specify the value of `model_options.random_state` in your project's parameters: +```bash +kedro run --params random=3 +``` +You can also specify a default value to be used in case the runtime parameter is not specified with the `kedro run` command. Consider this catalog entry: +```yaml +companies: + type: pandas.CSVDataset + filepath: "${runtime_params:folder, 'data/01_raw'}/companies.csv" +``` +If the `folder` parameter is not passed through the CLI `--params` option with `kedro run`, the default value `'data/01_raw/'` is used for the `filepath`. ### How to use resolvers in the `OmegaConfigLoader` Instead of hard-coding values in your configuration files, you can also dynamically compute them using [`OmegaConf`'s @@ -211,63 +254,6 @@ CONFIG_LOADER_ARGS = { } } ``` - - - - -### How to bypass the configuration loading rules -You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. - -For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). - -Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: - -```{code-block} python -:lineno-start: 10 -:emphasize-lines: 8 - -from kedro.config import OmegaConfigLoader -from kedro.framework.project import settings - -conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = OmegaConfigLoader(conf_source=conf_path) - -# Bypass configuration patterns by setting the key and values directly on the config loader instance. -conf_loader["catalog"] = {"catalog_config": "something_new"} -``` - - - - -### How to use a custom configuration loader -You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: - -```python -from kedro.config import AbstractConfigLoader - - -class CustomConfigLoader(AbstractConfigLoader): - def __init__( - self, - conf_source: str, - env: str = None, - runtime_params: Dict[str, Any] = None, - ): - super().__init__( - conf_source=conf_source, env=env, runtime_params=runtime_params - ) - - # Custom implementation -``` -To use this custom configuration loader, set it as the project configuration loader in `src//settings.py` as follows: -```python -from package_name.custom_configloader import CustomConfigLoader - -CONFIG_LOADER_CLASS = CustomConfigLoader -``` - - - ### How to load credentials through environment variables The [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) enables you to load credentials from environment variables. To achieve this you have to use the `OmegaConfigLoader` and the `omegaconf` [`oc.env` resolver](https://omegaconf.readthedocs.io/en/2.3_branch/custom_resolvers.html#oc-env). You can use the `oc.env` resolver to access credentials from environment variables in your `credentials.yml`: From dbb2d9a7425324a61a84ce5b1e4dca9fc4a2807e Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 06:55:35 +0000 Subject: [PATCH 08/17] Fix the header Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 7cb0f15f0a..cebd867b8f 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -17,7 +17,7 @@ This page also contains a set of guidance for advanced configuration requirement * [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader) -## How to use a custom configuration loader +### How to use a custom configuration loader You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class: ```python From e766d4ecc86c41bc9619d034e1e067b178035905 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 08:52:45 +0000 Subject: [PATCH 09/17] add example for standalone config Signed-off-by: Nok --- .../configuration/advanced_configuration.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index cebd867b8f..0619cbea13 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -3,6 +3,7 @@ The documentation on [configuration](./configuration_basics.md) describes how to By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class. +## Advanced Configuration for Kedro Project This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects: * [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) @@ -289,3 +290,58 @@ CONFIG_LOADER_ARGS = { If no merge strategy is defined, the default destructive strategy will be applied. Note that this merge strategy setting only applies to configuration files in **different** environments. When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys. + + +## Advance Configurations without Kedro Project +By defaults, Kedro Project has a `base` and `local` environments. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embrace [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). + +Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). + +### Read configuration +The config loader can work with just a single file. +```sh +└── parameters.yml +``` + +```yml +# parameters.yml +learning_rate: 0.01 +train_test_ratio: 0.7 +``` + +```python +from kedro.config import OmegaConfigLoader +config_loader = OmegaConfigLoader(conf_source=".") + +# Optionally, you can also use environments +# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local") + +>>> config_loader["parameters"] +{'learning_rate': 0.01, 'train_test_ratio': 0.7} +``` + +For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration] + +### How to use Custom Resolvers with `OmegaConfigLoader` +You can register custom resolvers to use non-primitive type for parmaeters. + +```yml +# parameters.yml +polar_float64: "${polars: Float64}" +today: "${today:}" +``` + +```python +import polars as pl +from datetime import date + +from kedro.config import OmegaConfigLoader + +custom_resolvers = {"polars": lambda x: getattr(pl, x), + "today": lambda: date.today()} + +# Register custom resolvers +config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers) +>>> print(config_loader["parameters"]) +{'polar_float64': Float64, 'today': datetime.date(2023, 11, 23)} +``` From 9695601535e755e04656bdeff2b89bb17ce9f83c Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Thu, 23 Nov 2023 22:36:03 +0800 Subject: [PATCH 10/17] Apply suggestions from code review Co-authored-by: Jo Stichbury Signed-off-by: Nok Lam Chan --- docs/source/configuration/advanced_configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 0619cbea13..7fefc89750 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -3,7 +3,7 @@ The documentation on [configuration](./configuration_basics.md) describes how to By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class. -## Advanced Configuration for Kedro Project +## Advanced configuration for Kedro projects This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects: * [How to use a custom config loader](#how-to-use-a-custom-configuration-loader) @@ -75,7 +75,7 @@ CONFIG_LOADER_ARGS = { ### How to bypass the configuration loading rules You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration. -For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md##use-hooks-to-load-external-credentials). +For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials). Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: @@ -292,8 +292,8 @@ If no merge strategy is defined, the default destructive strategy will be applie When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys. -## Advance Configurations without Kedro Project -By defaults, Kedro Project has a `base` and `local` environments. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embrace [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). +## Advanced configuration without a full Kedro project +By default, a Kedro project has a `base` and `local` environment. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embraces [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). From 14b85be81606943aeeefec57f7999e2e48650358 Mon Sep 17 00:00:00 2001 From: Nok Lam Chan Date: Thu, 23 Nov 2023 22:36:23 +0800 Subject: [PATCH 11/17] Update docs/source/configuration/configuration_basics.md Co-authored-by: Jo Stichbury Signed-off-by: Nok Lam Chan --- docs/source/configuration/configuration_basics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index 1851243017..ce78767f0b 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -22,7 +22,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con ## Configuration environments A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments. -By default, Kedro Framework has a `base` and a `local` environment. +By default, Kedro projects have a `base` and a `local` environment. ### Base In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments. From 8dd86c2b07a29aa85498bc499bd997676731b949 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 14:38:24 +0000 Subject: [PATCH 12/17] fix grammar Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 7fefc89750..2fb227bc01 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -293,7 +293,7 @@ When files are part of the same environment, they are always merged in a soft wa ## Advanced configuration without a full Kedro project -By default, a Kedro project has a `base` and `local` environment. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embraces [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). +By default, a Kedro project have a `base` and `local` environment. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embraces [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). From 483a7c9455a1deafa18a4fc680d0858f822e1d94 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 14:45:12 +0000 Subject: [PATCH 13/17] fix format Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 2fb227bc01..0d6f8d6fc9 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -293,17 +293,18 @@ When files are part of the same environment, they are always merged in a soft wa ## Advanced configuration without a full Kedro project -By default, a Kedro project have a `base` and `local` environment. In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. This is possible because Kedro embraces [modularity at the core](https://github.com/kedro-org/kedro/wiki/Kedro-Principles#1-modularity-at-the-core-%EF%B8%8F). + In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project have a `base` and `local` environment. Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). ### Read configuration -The config loader can work with just a single file. +The config loader can work without a Kedro project structure. ```sh +tree . └── parameters.yml ``` -```yml +```yaml # parameters.yml learning_rate: 0.01 train_test_ratio: 0.7 From a3ed825fd8d80ec3c505f8e4d2cd177d08f7af10 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 23 Nov 2023 14:48:56 +0000 Subject: [PATCH 14/17] fix format Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 0d6f8d6fc9..5af825ffdb 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -299,8 +299,9 @@ Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assume ### Read configuration The config loader can work without a Kedro project structure. -```sh +```bash tree . +. └── parameters.yml ``` @@ -321,12 +322,12 @@ config_loader = OmegaConfigLoader(conf_source=".") {'learning_rate': 0.01, 'train_test_ratio': 0.7} ``` -For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration] +For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md) ### How to use Custom Resolvers with `OmegaConfigLoader` You can register custom resolvers to use non-primitive type for parmaeters. -```yml +```yaml # parameters.yml polar_float64: "${polars: Float64}" today: "${today:}" From 55c812b22ae74db0f914c5365478cdee482aca8e Mon Sep 17 00:00:00 2001 From: Nok Date: Tue, 5 Dec 2023 10:05:17 +0000 Subject: [PATCH 15/17] fix grammar Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 5af825ffdb..2ed89a36eb 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -293,9 +293,8 @@ When files are part of the same environment, they are always merged in a soft wa ## Advanced configuration without a full Kedro project - In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project have a `base` and `local` environment. - -Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). + In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project has a `base` and `local` environment. +However, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md). ### Read configuration The config loader can work without a Kedro project structure. @@ -325,7 +324,7 @@ config_loader = OmegaConfigLoader(conf_source=".") For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md) ### How to use Custom Resolvers with `OmegaConfigLoader` -You can register custom resolvers to use non-primitive type for parmaeters. +You can register custom resolvers to use non-primitive types for parmaeters. ```yaml # parameters.yml From f8a910fd1652acd86ae6e766d557c225780608e9 Mon Sep 17 00:00:00 2001 From: Nok Date: Tue, 5 Dec 2023 10:14:19 +0000 Subject: [PATCH 16/17] missing plurals Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 2ed89a36eb..71fedb791b 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -77,7 +77,7 @@ You can bypass the configuration patterns and set configuration directly on the For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials). -Alternatively, if you are using config loader as a standalone component, you can override configuration as follow: +Alternatively, if you are using config loader as a standalone component, you can override configuration as follows: ```{code-block} python :lineno-start: 10 From a93678024d9529bde9bd0105245faefc0640a523 Mon Sep 17 00:00:00 2001 From: Nok Date: Thu, 7 Dec 2023 10:04:03 +0000 Subject: [PATCH 17/17] minor fix base on comments Signed-off-by: Nok --- docs/source/configuration/advanced_configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 07535b33df..2ad41ce6f7 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -349,11 +349,11 @@ config_loader = OmegaConfigLoader(conf_source=".") For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md) ### How to use Custom Resolvers with `OmegaConfigLoader` -You can register custom resolvers to use non-primitive types for parmaeters. +You can register custom resolvers to use non-primitive types for parameters. ```yaml # parameters.yml -polar_float64: "${polars: Float64}" +polars_float64: "${polars: Float64}" today: "${today:}" ``` @@ -369,5 +369,5 @@ custom_resolvers = {"polars": lambda x: getattr(pl, x), # Register custom resolvers config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers) >>> print(config_loader["parameters"]) -{'polar_float64': Float64, 'today': datetime.date(2023, 11, 23)} +{'polars_float64': Float64, 'today': datetime.date(2023, 11, 23)} ```