Created new QS guide to Add sources, staging and business-defined entities #6848
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
github-actions
bot
added
content
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| 7. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. | ||
|
|
||
| ### Add business-defined entities |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Add business-defined entities' should use sentence-style capitalization. Try 'Add business defined entities' instead.
| @@ -0,0 +1,280 @@ | |||
| ### Add sources | |||
There was a problem hiding this comment.
I suggest we configure these individual sections as H2 headers so they'll be separate pages within the guide. This might help alleviate scroll fatigue
Suggested change
| ### Add sources | |
| ## Add sources |
|
|
||
| --- | ||
|
|
||
| ## Why does structure matter? |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Why does structure matter?' should use sentence-style capitalization. Try 'Why does structure matter' instead.
|
|
||
| We'll use an analogy for working with dbt throughout this guide: thinking modularly in terms of atoms, molecules, and more complex outputs like proteins or cells (we apologize in advance to any chemists or biologists for our inevitable overstretching of this metaphor). Within that framework, if our source system data is a soup of raw energy and quarks, then you can think of the staging layer as condensing and refining this material into the individual atoms we’ll later build more intricate and useful structures with. | ||
|
|
||
| ## Staging: Files and folders |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Staging: Files and folders' should use sentence-style capitalization. Try 'Staging files and folders' instead.
| - **Folders.** Folder structure is extremely important in dbt. Not only do we need a consistent structure to find our way around the codebase, as with any software project, but our folder structure is also one of the key interfaces for understanding the knowledge graph encoded in our project (alongside the DAG and the data output into our warehouse). It should reflect how the data flows, step-by-step, from a wide variety of source-conformed models into fewer, richer business-conformed models. Moreover, we can use our folder structure as a means of selection in dbt [selector syntax](https://docs.getdbt.com/reference/node-selection/syntax). For example, with the above structure, if we got fresh Stripe data loaded and wanted to run all the models that build on our Stripe data, we can easily run `dbt build --select staging.stripe+` and we’re all set for building more up-to-date reports on payments. | ||
| - ✅ **Subdirectories based on the source system**. Our internal transactional database is one system, the data we get from Stripe's API is another, and lastly the events from our Snowplow instrumentation. We've found this to be the best grouping for most companies, as source systems tend to share similar loading methods and properties between tables, and this allows us to operate on those similar sets easily. | ||
| - ❌ **Subdirectories based on loader.** Some people attempt to group by how the data is loaded (Fivetran, Stitch, custom syncs), but this is too broad to be useful on a project of any real size. | ||
| - ❌ **Subdirectories based on business grouping.** Another approach we recommend against is splitting up by business groupings in the staging layer, and creating subdirectories like 'marketing', 'finance', etc. A key goal of any great dbt project should be establishing a single source of truth. By breaking things up too early, we open ourselves up to creating overlap and conflicting definitions (think marketing and financing having different fundamental tables for orders). We want everybody to be building with the same set of atoms, so in our experience, starting our transformations with our staging structure reflecting the source system structures is the best level of grouping for this step. |
There was a problem hiding this comment.
[custom.LatinAbbreviations] Avoid Latin abbreviations: 'and so on'. Consider using 'etc' instead.
| - ❌ `stg_[entity].sql` - might be specific enough at first, but will break down in time. Adding the source system into the file name aids in discoverability, and allows understanding where a component model came from even if you aren't looking at the file tree. | ||
| - ✅ **Plural.** SQL, and particularly SQL in dbt, should read as much like prose as we can achieve. We want to lean into the broad clarity and declarative nature of SQL when possible. As such, unless there’s a single order in your `orders` table, plural is the correct way to describe what is in a table with multiple rows. | ||
|
|
||
| ### Staging: Models |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Staging: Models' should use sentence-style capitalization. Try 'Staging models' instead.
| - Based on the above, the most standard types of staging model transformations are: | ||
| - ✅ **Renaming** | ||
| - ✅ **Type casting** | ||
| - ✅ **Basic computations** (e.g. cents to dollars) |
There was a problem hiding this comment.
[custom.LatinAbbreviations] Avoid Latin abbreviations: 'for example'. Consider using 'e.g' instead.
| └── customers.sql | ||
| ``` | ||
|
|
||
| ✅ **Group by department or area of concern.** If you have fewer than 10 or so marts you may not have much need for subfolders, so as with the intermediate layer, don’t over-optimize too early. If you do find yourself needing to insert more structure and grouping though, use useful business concepts here. In our marts layer, we’re no longer worried about source-conformed data, so grouping by departments (marketing, finance, etc.) is the most common structure at this stage. |
There was a problem hiding this comment.
[custom.LatinAbbreviations] Avoid Latin abbreviations: 'and so on'. Consider using 'etc' instead.
|
|
||
| ❌ **Build the same concept differently for different teams.** `finance_orders` and `marketing_orders` is typically considered an anti-pattern. There are, as always, exceptions — a common pattern we see is that, finance may have specific needs, for example reporting revenue to the government in a way that diverges from how the company as a whole measures revenue day-to-day. Just make sure that these are clearly designed and understandable as _separate_ concepts, not departmental views on the same concept: `tax_revenue` and `revenue` not `finance_revenue` and `marketing_revenue`. | ||
|
|
||
| ### Marts: Models |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Marts: Models' should use sentence-style capitalization. Try 'Marts models' instead.
| The most important aspect of marts is that they contain all of the useful data about a _particular entity_ at a granular level. That doesn’t mean we don’t bring in lots of other entities and concepts, like tons of `user` data into our `orders` mart, we do! It just means that individual `orders` remain the core grain of our table. If we start grouping `users` and `orders` along a [date spine](https://github.com/dbt-labs/dbt-utils#date_spine-source), into something like `user_orders_per_day`, we’re moving past marts into _metrics_. | ||
| ::: | ||
|
|
||
| ### Marts: Other considerations |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'Marts: Other considerations' should use sentence-style capitalization. Try 'Marts other considerations' instead.
|
|
||
| - **Troubleshoot via tables.** While stacking views and ephemeral models up until our marts — only building data into the warehouse at the end of a chain when we have the models we really want end users to work with — is ideal in production, it can present some difficulties in development. Particularly, certain errors may seem to be surfacing in our later models that actually stem from much earlier dependencies in our model chain (ancestor models in our DAG that are built before the model throws the errors). If you’re having trouble pinning down where or what a database error is telling you, it can be helpful to temporarily build a specific chain of models as tables so that the warehouse will throw the error where it’s actually occurring. | ||
|
|
||
| ### The dbt Semantic Layer and marts |
There was a problem hiding this comment.
[custom.SentenceCaseHeaders] 'The dbt Semantic Layer and marts' should use sentence-style capitalization. Try 'The dbt semantic layer and marts' instead.
| - ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). | ||
| - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). | ||
| - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. | ||
| - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. |
There was a problem hiding this comment.
[custom.LatinAbbreviations] Avoid Latin abbreviations: 'that is'. Consider using 'idea' instead.
github-actions
bot
added
size: small
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What are you changing in this pull request and why?
I have created this PR following this git issue: #2978
The Git issue advises the BQ QS uses models as opposed to sources in the model, which isn't entirely wrong but may be an example of bad habits. Following this, I found the RS and DB QSs were also missing sources, staging and business entities so I have updated the QSs to show this.
Closes: #2978
Checklist
🚀 Deployment available! Here are the direct links to the updated files: