Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentations of Configuration default environments #3321

Merged
merged 21 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 63 additions & 1 deletion docs/source/configuration/advanced_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ The documentation on [configuration](./configuration_basics.md) describes how to

By default, Kedro is set up to use the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader) class.

## Advanced configuration for Kedro projects
This page also contains a set of guidance for advanced configuration requirements of standard Kedro projects:

* [How to use a custom config loader](#how-to-use-a-custom-configuration-loader)
Expand All @@ -17,7 +18,7 @@ This page also contains a set of guidance for advanced configuration requirement
* [How to change the merge strategy used by `OmegaConfigLoader`](#how-to-change-the-merge-strategy-used-by-omegaconfigloader)


## How to use a custom configuration loader
### How to use a custom configuration loader
You can implement a custom configuration loader by extending the [`AbstractConfigLoader`](/kedro.config.AbstractConfigLoader) class:

```python
Expand Down Expand Up @@ -74,6 +75,10 @@ CONFIG_LOADER_ARGS = {
### How to bypass the configuration loading rules
You can bypass the configuration patterns and set configuration directly on the instance of a config loader class. You can bypass the default configuration (catalog, parameters and credentials) as well as additional configuration.

For example, you can [use hooks to load external credentials](../hooks/common_use_cases.md#use-hooks-to-load-external-credentials).

Alternatively, if you are using config loader as a standalone component, you can override configuration as follow:
merelcht marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} python
:lineno-start: 10
:emphasize-lines: 8
Expand Down Expand Up @@ -285,3 +290,60 @@ CONFIG_LOADER_ARGS = {

If no merge strategy is defined, the default destructive strategy will be applied. Note that this merge strategy setting only applies to configuration files in **different** environments.
When files are part of the same environment, they are always merged in a soft way. An error is thrown when files in the same environment contain the same top-level keys.


## Advanced configuration without a full Kedro project
In some cases, you may only want to use the `OmegaConfigLoader` without a Kedro project. By default, a Kedro project have a `base` and `local` environment.
merelcht marked this conversation as resolved.
Show resolved Hide resolved

merelcht marked this conversation as resolved.
Show resolved Hide resolved
Unlike a Kedro Project, when you use the `OmegaConfigLoader` directly, it assumes *no* environment. You may find it useful to [add Kedro to your existing notebooks](../notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md).
merelcht marked this conversation as resolved.
Show resolved Hide resolved

### Read configuration
The config loader can work without a Kedro project structure.
```bash
tree .
.
└── parameters.yml
```

```yaml
# parameters.yml
learning_rate: 0.01
train_test_ratio: 0.7
```

```python
from kedro.config import OmegaConfigLoader
config_loader = OmegaConfigLoader(conf_source=".")

# Optionally, you can also use environments
# config_loader = OmegaConfigLoader(conf_source=".", base_env="base", default_run_env="local")

>>> config_loader["parameters"]
{'learning_rate': 0.01, 'train_test_ratio': 0.7}
```

For the full list of features, please refer to [configuration_basics](./configuration_basics.md) and [advanced_configuration](./advanced_configuration.md)

### How to use Custom Resolvers with `OmegaConfigLoader`
You can register custom resolvers to use non-primitive type for parmaeters.
merelcht marked this conversation as resolved.
Show resolved Hide resolved

```yaml
# parameters.yml
polar_float64: "${polars: Float64}"
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved
today: "${today:}"
```

```python
import polars as pl
from datetime import date

from kedro.config import OmegaConfigLoader

custom_resolvers = {"polars": lambda x: getattr(pl, x),
"today": lambda: date.today()}

# Register custom resolvers
config_loader = OmegaConfigLoader(conf_source=".", custom_resolvers=custom_resolvers)
>>> print(config_loader["parameters"])
{'polar_float64': Float64, 'today': datetime.date(2023, 11, 23)}
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved
```
4 changes: 2 additions & 2 deletions docs/source/configuration/configuration_basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The configuration source folder is [`conf`](../get_started/kedro_concepts.md#con
## Configuration environments
A configuration environment is a way of organising your configuration settings for different stages of your data pipeline. For example, you might have different settings for development, testing, and production environments.

By default, Kedro has a `base` and a `local` environment.
By default, Kedro projects have a `base` and a `local` environment.

### Base
In Kedro, the base configuration environment refers to the default configuration settings that are used as the foundation for all other configuration environments.
Expand Down Expand Up @@ -60,7 +60,7 @@ Configuration files will be matched according to file name and type rules. Suppo
* *Either* of the following is true:
* filename starts with `catalog`
* file is located in a subfolder whose name is prefixed with `catalog`
* *And* file extension is one of the following: yaml`, `yml`, or `json`
* *And* file extension is one of the following: `yaml`, `yml`, or `json`

### Configuration patterns
Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@
"source": [
"from kedro.config import OmegaConfigLoader\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")"
"conf_loader = OmegaConfigLoader(conf_source=\".\")"
]
},
{
Expand Down Expand Up @@ -548,7 +548,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"\n",
"# Create the DataCatalog instance from the configuration\n",
Expand Down Expand Up @@ -584,7 +584,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"conf_params = conf_loader[\"parameters\"]\n",
"\n",
Expand Down Expand Up @@ -804,7 +804,7 @@
"from kedro.config import OmegaConfigLoader\n",
"from kedro.io import DataCatalog\n",
"\n",
"conf_loader = OmegaConfigLoader(\".\", base_env=\"\", default_run_env=\"\")\n",
"conf_loader = OmegaConfigLoader(conf_source=\".\")\n",
"conf_catalog = conf_loader[\"catalog\"]\n",
"conf_params = conf_loader[\"parameters\"]\n",
"\n",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ To use Kedro's `OmegaConfigLoader` to load `parameters.yml` the code is as follo
```python
from kedro.config import OmegaConfigLoader
conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
```

```python
Expand Down Expand Up @@ -348,7 +348,7 @@ To load `catalog.yml` the code is as follows:
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog
conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
# Create the DataCatalog instance from the configuration
Expand All @@ -372,7 +372,7 @@ Code in a Kedro project runs in one or more pipelines, where a pipeline is a ser
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog
conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]
Expand Down Expand Up @@ -546,7 +546,7 @@ And that's it. The notebook code has been refactored into a series of functions.
from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog
conf_loader = OmegaConfigLoader(".", base_env="", default_run_env="")
conf_loader = OmegaConfigLoader(conf_source=".")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]
Expand Down