Update documentations of Configuration default environments (#3321)

* update docs Signed-off-by: Nok <[email protected]> * add docs Signed-off-by: Nok <[email protected]> * bubble up the common features and fix some level of headers Signed-off-by: Nok <[email protected]> * update notebook with keywords Signed-off-by: Nok <[email protected]> * remove default environments from notebook Signed-off-by: Nok <[email protected]> * typo fix Signed-off-by: Nok <[email protected]> * Revert "bubble up the common features and fix some level of headers" This reverts commit e6b9587. * Fix the header Signed-off-by: Nok <[email protected]> * add example for standalone config Signed-off-by: Nok <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Nok Lam Chan <[email protected]> * Update docs/source/configuration/configuration_basics.md Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Nok Lam Chan <[email protected]> * fix grammar Signed-off-by: Nok <[email protected]> * fix format Signed-off-by: Nok <[email protected]> * fix format Signed-off-by: Nok <[email protected]> * fix grammar Signed-off-by: Nok <[email protected]> * missing plurals Signed-off-by: Nok <[email protected]> --------- Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Lam Chan <[email protected]> Co-authored-by: Jo Stichbury <[email protected]>
kedro-org · Dec 7, 2023 · 83eed17 · 83eed17
1 parent 073d6b8
commit 83eed17
Show file tree

Hide file tree

Showing 4 changed files with 72 additions and 11 deletions.
diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md
@@ -3,6 +3,7 @@ The documentation on [configuration](./configuration_basics.md) describes how to
 
 By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class.
 
+## Advanced configuration for Kedro projects
 This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects:
 
 * [How to use a custom config loader](#how-to-use-a-custom-configuration-loader)
@@ -18,7 +19,7 @@ This page also contains a set of guidance for advanced configuration requirement
 * [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader)
 
 
-## How to use a custom configuration loader
+### How to use a custom configuration loader
 You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class:
 
 ```python
@@ -75,6 +76,10 @@ CONFIG_LOADER_ARGS = {
 ### How to bypass the configuration loading rules
 You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration.
 
+For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials).
+
+Alternatively, if you are using config loader as a standalone component, you can override configuration as follows:
+
 ```{code-block} python
 :lineno-start: 10
 :emphasize-lines: 8
@@ -310,3 +315,59 @@ CONFIG_LOADER_ARGS = {
 
 If no merge strategy is defined, the default destructive strategy will be applied. Note that this merge strategy setting only applies to configuration files in **different** environments.
 When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys.
+
+
+## Advanced configuration without a full Kedro project
+ In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project has a `base` and `local` environment.
+However, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md).
+
+### Read configuration
+The config loader can work without a Kedro project structure.
+```bash
+tree .
+.
+└── parameters.yml
+```
+
+```yaml
+# parameters.yml
+learning_rate: 0.01
+train_test_ratio: 0.7
+```
+
+```python
+from kedro.config import OmegaConfigLoader
+config_loader = OmegaConfigLoader(conf_source=".")
+
+# Optionally, you can also use environments
+# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local")
+
+>>> config_loader["parameters"]
+{'learning_rate': 0.01, 'train_test_ratio': 0.7}
+```
+
+For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md)
+
+### How to use Custom Resolvers with `OmegaConfigLoader`
+You can register custom resolvers to use non-primitive types for parameters.
+
+```yaml
+# parameters.yml
+polars_float64: "${polars: Float64}"
+today: "${today:}"
+```
+
+```python
+import polars as pl
+from datetime import date
+
+from kedro.config import OmegaConfigLoader
+
+custom_resolvers = {"polars": lambda x: getattr(pl, x),
+                    "today": lambda: date.today()}
+
+# Register custom resolvers
+config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers)
+>>> print(config_loader["parameters"])
+{'polars_float64': Float64, 'today': datetime.date(2023, 11, 23)}
+```
diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md
@@ -44,7 +44,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con
 ## Configuration environments
 A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments.
 
-By default, Kedro has a `base` and a `local` environment.
+By default, Kedro projects have a `base` and a `local` environment.
 
 ### Base
 In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments.
@@ -82,7 +82,7 @@ Configuration files will be matched according to file name and type rules. Suppo
 * *Either* of the following is true:
   * filename starts with `catalog`
   * file is located in a subfolder whose name is prefixed with `catalog`
-* *And* file extension is one of the following: yaml`, `yml`, or `json`
+* *And* file extension is one of the following: `yaml`, `yml`, or `json`
 
 ### Configuration patterns
 Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes.

diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb
@@ -436,7 +436,7 @@
    "source": [
     "from kedro.config import OmegaConfigLoader\n",
     "\n",
-    "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")"
+    "conf_loader = OmegaConfigLoader(conf_source=\".\")"
    ]
   },
   {
@@ -548,7 +548,7 @@
     "from kedro.config import OmegaConfigLoader\n",
     "from kedro.io import DataCatalog\n",
     "\n",
-    "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
+    "conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
     "conf_catalog = conf_loader[\"catalog\"]\n",
     "\n",
     "# Create the DataCatalog instance from the configuration\n",
@@ -584,7 +584,7 @@
     "from kedro.config import OmegaConfigLoader\n",
     "from kedro.io import DataCatalog\n",
     "\n",
-    "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
+    "conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
     "conf_catalog = conf_loader[\"catalog\"]\n",
     "conf_params = conf_loader[\"parameters\"]\n",
     "\n",
@@ -804,7 +804,7 @@
     "from kedro.config import OmegaConfigLoader\n",
     "from kedro.io import DataCatalog\n",
     "\n",
-    "conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
+    "conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
     "conf_catalog = conf_loader[\"catalog\"]\n",
     "conf_params = conf_loader[\"parameters\"]\n",
     "\n",

diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md
@@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo
 ```python
 from kedro.config import OmegaConfigLoader
 
-conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
+conf_loader = OmegaConfigLoader(conf_source=".")
 ```
 
 ```python
@@ -348,7 +348,7 @@ To load `catalog.yml` the code is as follows:
 from kedro.config import OmegaConfigLoader
 from kedro.io import DataCatalog
 
-conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
+conf_loader = OmegaConfigLoader(conf_source=".")
 conf_catalog = conf_loader["catalog"]
 
 # Create the DataCatalog instance from the configuration
@@ -372,7 +372,7 @@ Code in a Kedro project runs in one or more pipelines, where a pipeline is a ser
 from kedro.config import OmegaConfigLoader
 from kedro.io import DataCatalog
 
-conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
+conf_loader = OmegaConfigLoader(conf_source=".")
 conf_catalog = conf_loader["catalog"]
 conf_params = conf_loader["parameters"]
 
@@ -546,7 +546,7 @@ And that's it. The notebook code has been refactored into a series of functions.
 from kedro.config import OmegaConfigLoader
 from kedro.io import DataCatalog
 
-conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
+conf_loader = OmegaConfigLoader(conf_source=".")
 conf_catalog = conf_loader["catalog"]
 conf_params = conf_loader["parameters"]