-
Notifications
You must be signed in to change notification settings - Fork 54
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
5 changed files
with
220 additions
and
139 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,87 +1,60 @@ | ||
| --- | ||
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| # Configuration | ||
|
|
||
| ## Configuring dbt-osmosis | ||
|
|
||
| ### Models | ||
|
|
||
| dbt-osmosis' primary purpose is to automatically generate and manage YAML files for your dbt models. We opt for explicitness over implicitness. Thus the following configuration is required to even run dbt-osmosis. By specifying this configuration at the top-level beneath your project key, you are specifying a default configuration for all models in your project. You can override this configuration for individual models by specifying the `+dbt-osmosis` configuration at various levels of the hierarchy in the configuration file. These levels match your folder structure exactly. | ||
| At minimum, each **folder** (or subfolder) of models in your dbt project must specify the `+dbt-osmosis` directive so that dbt-osmosis knows **where** to create or move the YAML files. | ||
|
|
||
| ```yaml title="dbt_project.yml" | ||
| models: | ||
| <your_project_name>: | ||
| +dbt-osmosis: <path> | ||
| ``` | ||
| - `<your_project_name>` is the name of your dbt project. | ||
| - `<path>` is the path to the YAML file that will be generated for the model. This path is **relative to the model's (sql file) directory.** | ||
|
|
||
| #### Examples | ||
|
|
||
| ```yaml title="dbt_project.yml" | ||
| models: | ||
| your_project_name: | ||
| # a default blanket rule | ||
| +dbt-osmosis: "_{model}.yml" | ||
| +dbt-osmosis: "_{model}.yml" # Default for entire project | ||
|
|
||
| staging: | ||
| # nest docs in subfolder relative to model | ||
| +dbt-osmosis: "schema/{model}.yml" | ||
| +dbt-osmosis: "{parent}.yml" # Each subfolder lumps docs by folder name | ||
|
|
||
| intermediate: | ||
| # separate docs based on materialization | ||
| # Example of using node.config or node.tags | ||
| +dbt-osmosis: "{node.config[materialized]}/{model}.yml" | ||
|
|
||
| marts: | ||
| # static paths are perfectly fine! | ||
| # A single schema file for all models in 'marts' | ||
| +dbt-osmosis: "prod.yml" | ||
| ``` | ||
|
|
||
| ### Sources | ||
|
|
||
| dbt-osmosis can be configured to automatically generate YAML files for your dbt sources. To enable this feature, add the following to your `dbt_project.yml` file. | ||
|
|
||
| ```yaml title="dbt_project.yml" | ||
| vars: | ||
| dbt-osmosis: | ||
| <source_name>: <path> | ||
| <source_name>: | ||
| path: <path> | ||
| database: <database> | ||
| schema: <schema> | ||
| _blacklist: <blacklist> | ||
| seeds: | ||
| <your_project_name>: | ||
| +dbt-osmosis: "_schema.yml" | ||
| ``` | ||
| - `<source_name>` is the name of a source in your `dbt_project.yml` file. | ||
| - `<path>` is the path to the YAML file that will be generated for the source. This path is relative to the root of your dbt project models directory. | ||
| - `<database>` is the database that will be used for the source. If not specified, the database will default to the one in your profiles.yml file. | ||
| - `<schema>` is the schema that will be used for the source. If not specified, the source name is assumed to be the schema which matches dbt's default behavior. | ||
| - `<blacklist>` is the columns to be ignored. You can use regular expressions to specify which columns you'd like to exclude. | ||
| ### Sources | ||
| #### Examples | ||
| You can optionally configure dbt-osmosis to manage sources automatically. In your `dbt_project.yml`: | ||
|
|
||
| ```yaml title="dbt_project.yml" | ||
| vars: | ||
| dbt-osmosis: | ||
| # a source with a different schema | ||
| salesforce: | ||
| path: "staging/salesforce/source.yml" | ||
| schema: "salesforce_v2" | ||
| # a source with the same schema as the source name | ||
| marketo: "staging/customer/marketo.yml" | ||
| sources: | ||
| salesforce: | ||
| path: "staging/salesforce/source.yml" | ||
| schema: "salesforce_v2" | ||
| # a special variable interpolated at runtime | ||
| jira: "staging/project_mgmt/{parent}.yml" | ||
| marketo: "staging/customer/marketo.yml" | ||
| jira: "staging/project_mgmt/{parent}.yml" | ||
| github: "all_sources/github.yml" | ||
| # a dedicated directory for all sources | ||
| github: "all_sources/github.yml" | ||
| _blacklist: | ||
| # Columns matching these patterns will be ignored (like ephemeral system columns) | ||
| column_ignore_patterns: | ||
| - "_FIVETRAN_SYNCED" | ||
| - ".*__key__.namespace" | ||
| ``` | ||
|
|
||
| Notice the use of the `{parent}` variable in the `jira` source configuration. This variable is a special variable that will be replaced with the name of the parent directory of the YAML file. The other special variables are `{node}` and `{model}`. We will discuss these variables in more detail in the next section. | ||
| **Key points:** | ||
|
|
||
| - `vars: dbt-osmosis: sources: <source_name>` sets where the source YAML file should live. | ||
| - If the source does not actually exist yet, dbt-osmosis can bootstrap it. | ||
| - If you omit `schema`, dbt-osmosis infers it is the same as your source name. |
Oops, something went wrong.