Skip to content

Commit

Permalink
combine and simplify sq and exports (#5470)
Browse files Browse the repository at this point in the history 
this pr consolidates the saved queries and exports docs so its clearer
how both work together. it also removes the comparison table and adds a
'configure exports' section in teh saved queries page, making sure they
flow correctly.

[notion
project](https://www.notion.so/dbtlabs/Consolidate-Save-queries-and-Exports-document-e5cc1a7ccd9541089a57c9d9ea5008f4?pvs=4)
  • Loading branch information
@mirnawong1
mirnawong1 authored May 16, 2024
2 parents ae3fb40 + 8fe8e4b commit f4be93b
Show file tree
Hide file tree
Showing 8 changed files with 322 additions and 153 deletions.
191 changes: 135 additions & 56 deletions website/docs/docs/build/saved-queries.md
View file

Large diffs are not rendered by default.

116 changes: 37 additions & 79 deletions website/docs/docs/use-dbt-semantic-layer/exports.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,88 +4,41 @@ description: "Use exports to write tables to the data platform on a schedule."
sidebar_label: "Write queries with exports"
---

The exports feature in the dbt Semantic Layer enhances the [saved queries](/docs/build/saved-queries) by allowing you to write commonly used queries directly within your data platform.
Exports enhance [saved queries](/docs/build/saved-queries) by running your saved queries and writing the output to a table or view within your data platform. Saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by:

While saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by:

- Enabling you to write these queries within your data platform using dbt Cloud.
- Enabling you to write these queries within your data platform using the dbt Cloud job scheduler.
- Proving an integration path for tools that don't natively support the dbt Semantic Layer by exposing tables of metrics and dimensions.

Essentially, exports are like any other table in your data platform. They enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class Semantic Layer integration. Refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more information.
Essentially, exports are like any other table in your data platform — they enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class [Semantic Layer integration](/docs/use-dbt-semantic-layer/avail-sl-integrations). Running an export counts towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. Querying the resulting table or view from the export does not count toward queried metric usage.

## Prerequisites

- You have a [multi-tenant](/docs/cloud/about-cloud/tenancy) dbt Cloud account on a [Team or Enterprise](https://www.getdbt.com/pricing/) plan. Single-tenant is not supported at this time.
- You have a [multi-tenant](/docs/cloud/about-cloud/tenancy) dbt Cloud account on a [Team or Enterprise](https://www.getdbt.com/pricing/) plan. Single-tenant isn't supported at this time.
- You use one of the following data platforms: Snowflake, BigQuery, Databricks, or Redshift.
- You are on [dbt version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) 1.7 or newer.
- You have the dbt Semantic Layer [configured](/docs/use-dbt-semantic-layer/setup-sl) in your dbt project.
- You have a dbt Cloud environment with a [Job scheduler](/docs/deploy/job-scheduler) enabled.

### Comparison between exports and saved queries


| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans with dbt versions 1.7 or newer. <br /> Note, Exports are not supported in dbt Cloud IDE yet. | Available in both dbt Core and dbt Cloud. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, including metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />Example: Create a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />Example: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |

## Define exports

Exports are an additional configuration added to a saved query. They define how to write a saved query, along with the schema and table name.

You can define `exports` in YAML format as a key within the `saved_queries` configuration and in the same file as your metric definitions.

An example of a saved query with an export:

<File name='semantic_model.yml'>

```yaml
saved_queries:
- name: order_metrics
description: Relevant order metrics
query_params:
metrics:
- orders
- large_order
- food_orders
- order_total
group_by:
- Entity('order_id')
- TimeDimension('metric_time', 'day')
- Dimension('customer__customer_name')
- ... # Additional group_by
where:
- "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'"
- ... # Additional where clauses
exports:
- name: order_metrics
config:
export_as: table # Options available: table, view
schema: YOUR_SCHEMA # Optional - defaults to deployment schema
alias: SOME_TABLE_NAME # Optional - defaults to Export name
```
</File>
- You have a dbt Cloud environment with the [job scheduler](/docs/deploy/job-scheduler) enabled.
- You have a [saved query](/docs/build/saved-queries) and [export configured](/docs/build/saved-queries#configure-exports) in your dbt project. In your configuration, leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) to cache common queries, speed up performance, and reduce compute costs.
- You have the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) installed. Note, that exports aren't supported in dbt Cloud IDE yet.

## Run exports

Once you define exports in your dbt project, then you can run them. There are two ways to run an export:
Before you're able to run exports in development or production, you'll need to make sure you've [configured saved queries and exports](/docs/build/saved-queries) in your dbt project. In your saved query config, you can also leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) with the dbt Cloud job scheduler to cache common queries, speed up performance, and reduce compute costs.

There are two ways to run an export:

- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation). Note, the dbt Cloud IDE doesn't support Exports yet.
- [Run exports in production](#exports-in-production) using the [dbt Cloud job scheduler](/docs/deploy/job-scheduler).
- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to test the output before production (dbt Cloud IDE isn't supported yet).
- [Run exports in production](#exports-in-production) using the [dbt Cloud job scheduler](/docs/deploy/job-scheduler) to write these queries within your data platform.

### Exports in development
## Exports in development

You can run an export in your development environment using your development credentials if you want to test the output of the export before production. You can use the following command to run exports in the dbt Cloud CLI:

```bash
dbt sl export
```

The following table lists the options for `dbt sl export` command:
The following table lists the options for `dbt sl export` command, using the `--` flag prefix to specify the parameters:

| Parameters | Type | Required | Description |
| ------- | --------- | ---------- | ---------------- |
Expand All @@ -99,20 +52,22 @@ The following table lists the options for `dbt sl export` command:

You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output:

**Example**
#### Example

```bash
dbt sl export --saved-query sq_name
```

**Output**
#### Output

```bash
Polling for export status - query_id: 2c1W6M6qGklo1LR4QqzsH7ASGFs..
Export completed.
```

### Use the select flag

By default, all exports are run for a saved query. You can use the `select` flag in [development](#exports-in-development).
You can have multiple exports for a saved query and by default, all exports are run for a saved query. You can use the `select` flag in [development](#exports-in-development) to select specific or multiple exports. Note, you can’t sub-select metrics or dimensions from the saved query, it’s just to change the export configuration i.e table format or schema

For example, the following command runs `export_1` and `export_2` and doesn't work with the `--alias` or `--export_as` flags:

Expand All @@ -138,14 +93,16 @@ dbt sl export --saved-query sq_number1 --export-as table --alias new_export
```
</details>

### Exports in production
## Exports in production

Enabling and executing exports in dbt Cloud optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions.

To enable exports in production using the dbt Cloud job scheduler:
To enable exports in production to run saved queries and write them within your data platform, you'll need to set up dbt Cloud job scheduler and perform the following steps:

1. [Set environment variable](#set-environment-variable)
1. [Create and execute exports](#create-and-execute-exports)
1. [Set an environment variable](#set-environment-variable) in dbt Cloud.
2. [Create and execute export](#create-and-execute-exports) job run.

#### Set environment variable
### Set environment variable
<!-- for version 1.7 -->
<VersionBlock firstVersion lastVersion="1.7">

Expand Down Expand Up @@ -178,32 +135,32 @@ If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_EXPORT_SAVED

When you run a build job, any saved queries downstream of the dbt models in that job will also run. To make sure your export data is up-to-date, run the export as a downstream step (after the model).

#### Create and execute exports
### Create and execute exports
<VersionBlock firstVersion lastVersion="1.7">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_INCLUDE_SAVED_QUERY=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.
- Use the [selector syntax](/reference/node-selection/syntax) `--select` or `-s` option in your build command to specify a particular dbt model or saved query to run. For example, to run all saved queries downstream of the `orders` semantic model, use the following command:
```bash
dbt build --select orders+
```

</VersionBlock>

<VersionBlock firstVersion="1.8">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_EXPORT_SAVED_QUERIES=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.
- Use the [selector syntax](/reference/node-selection/syntax) `--select` or `-s` option in your build command to specify a particular dbt model or saved query to run. For example, to run all saved queries downstream of the `orders` semantic model, use the following command:
```bash
dbt build --select orders+
```

</VersionBlock>

2. After dbt finishes building the models, the MetricFlow Server processes the exports, compiles the necessary SQL, and executes this SQL against your data platform. It directly executes a "create table" statement so the data stays within your data platform.
3. Review the exports' execution details in the jobs logs and confirm the export was run successfully. This helps troubleshoot and to ensure accuracy. Since saved queries are integrated into the dbt DAG, all outputs related to exports are available in the job logs.
4. Your data is now available in the data platform for querying.

Enabling and executing exports in dbt Cloud optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions.

You can use the [selector syntax](/reference/node-selection/syntax) `--select` or `-s` to specify a particular dbt model to run in your build command to only run the exports downstream of that model, or to select a saved query to run. As an example, the following command will run any saved queries that are downstream of the `orders` semantic model:

```bash
dbt build --select orders+
```
4. Your data is now available in the data platform for querying! 🎉
## FAQs
Expand Down Expand Up @@ -246,4 +203,5 @@ To select `saved_queries` by resource type, run `dbt build --resource-type saved
</detailsToggle>

## Related docs
- Configure [caching](/docs/use-dbt-semantic-layer/sl-cache)
- [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs)
7 changes: 5 additions & 2 deletions website/docs/reference/dbt_project.yml.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ The following example is a list of all available configurations in the `dbt_proj
identifier: true | false

metrics:
<metric-configs>
[<metric-configs>](/docs/build/metrics-overview)

models:
[<model-configs>](/reference/model-configs)
Expand All @@ -72,7 +72,10 @@ seeds:
[<seed-configs>](/reference/seed-configs)

semantic-models:
<semantic-model-configs>
[<semantic-model-configs>](/docs/build/semantic-models)

saved-queries:
[<saved-queries-configs>](/docs/build/saved-queries)

snapshots:
[<snapshot-configs>](/reference/snapshot-configs)
Expand Down
41 changes: 37 additions & 4 deletions website/docs/reference/resource-configs/enabled.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ default_value: true
{ label: 'Metrics', value: 'metrics', },
{ label: 'Exposures', value: 'exposures', },
{ label: 'Semantic models', value: 'semantic models', },
{ label: 'Saved queries', value: 'saved queries', },
]
}>
<TabItem value="models">
Expand Down Expand Up @@ -189,7 +190,6 @@ Support for disabling metrics was added in dbt Core v1.3
metrics:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)enabled: true | false

```
</File>
Expand All @@ -203,7 +203,6 @@ metrics:
- name: [<metric-name>]
[config](/reference/resource-properties/config):
enabled: true | false

```
</File>
Expand All @@ -228,7 +227,6 @@ Support for disabling exposures was added in dbt Core v1.3
exposures:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)enabled: true | false

```
</File>
Expand All @@ -242,7 +240,6 @@ exposures:
- name: [<exposure-name>]
[config](/reference/resource-properties/config):
enabled: true | false

```
</File>
Expand Down Expand Up @@ -286,9 +283,45 @@ semantic_models:
</TabItem>
<TabItem value="saved queries">
<VersionBlock lastVersion="1.6">
Support for disabling saved queries has been added in dbt Core v1.7.
</VersionBlock>
<VersionBlock firstVersion="1.7">
<File name='dbt_project.yml'>
```yaml
saved-queries:
[<resource-path>](/reference/resource-configs/resource-path):
[+](/reference/resource-configs/plus-prefix)enabled: true | false
```
</File>
<File name='models/semantic_models.yml'>
```yaml
saved_queries:
- name: [<saved_query_name>]
[config](/reference/resource-properties/config):
enabled: true | false
```
</File>
</VersionBlock>
</TabItem>
</Tabs>
## Definition
An optional configuration for enabling or disabling a resource.
* Default: true
Expand Down
37 changes: 34 additions & 3 deletions website/docs/reference/resource-configs/group.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ id: "group"
{ label: 'Analyses', value: 'analyses', },
{ label: 'Metrics', value: 'metrics', },
{ label: 'Semantic models', value: 'semantic models', },
{ label: 'Saved queries', value: 'saved queries',} ,
]
}>
<TabItem value="models">
Expand Down Expand Up @@ -280,25 +281,55 @@ Support for grouping semantic models has been added in dbt Core v1.7.
<File name='dbt_project.yml'>

```yaml

semantic-models:
[<resource-path>](resource-path):
[+](plus-prefix)group: GROUP_NAME

```
</File>
<File name='models/semantic_models.yml'>
```yaml

semantic_models:
- name: SEMANTIC_MODEL_NAME
config:
group: GROUP_NAME
```
</File>
</VersionBlock>
</TabItem>
<TabItem value="saved queries">
<VersionBlock lastVersion="1.6">
Support for grouping saved queries has been added in dbt Core v1.7.
</VersionBlock>
<VersionBlock firstVersion="1.7">
<File name='dbt_project.yml'>
```yaml
saved-queries:
[<resource-path>](resource-path):
[+](plus-prefix)group: GROUP_NAME
```
</File>
<File name='models/semantic_models.yml'>
```yaml
saved_queries:
- name: SAVED_QUERY_NAME
config:
group: GROUP_NAME
```
</File>
Expand Down
Loading

0 comments on commit f4be93b

Please sign in to comment.