Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding fct_unused_staging_models #523

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/img/unused_staging_models.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ hide:
|Modeling |[Root Models](../rules/modeling/#root-models) |`fct_root_models`|
|Modeling |[Staging Models Dependent on Downstream Models](../rules/modeling/#staging-models-dependent-on-downstream-models) |`fct_staging_dependent_on_marts_or_intermediate`|
|Modeling |[Unused Sources](../rules/modeling/#unused-sources) |`fct_unused_sources`|
|Modeling |[Unused Staging Models](../rules/modeling/#unused-staging-models) |`fct_unused_staging_models`|
|Modeling |[Models with Too Many Joins](../rules/modeling/#models-with-too-many-joins) |`fct_too_many_joins`|
|Testing |[Missing Primary Key Tests](../rules/testing/#missing-primary-key-tests) |`fct_missing_primary_key_tests`|
|Testing |[Missing Source Freshness](../rules/testing/#missing-source-freshness) |`fct_sources_without_freshness`|
Expand Down
20 changes: 20 additions & 0 deletions docs/rules/modeling.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,26 @@ or any other nested information.

![A refactored DAG showing three sources which are each being referenced by an accompanying staging model](https://user-images.githubusercontent.com/30663534/159603703-6e94b00b-07d1-4f47-89df-8e5685d9fcf0.png){ width=500 }

---
## Unused Staging Models

`fct_unused_staging_models` ([source](https://github.com/dbt-labs/dbt-project-evaluator/tree/main/models/marts/dag/fct_unused_staging_models.sql)) shows each staging model with 0 children.

**Example**

`model.stg_auditboard__teams` isn't being referenced.

![A DAG showing three staging_models which are each being referenced by an accompanying staging model, and one source that isn't being referenced at all](../img/unused_staging_models.png){ width=500 }

**Reason to Flag**

This represents a staging models that was created but never used in downstream models. This simply
represents the buildup of cruft in the project that doesn’t need to be there.

**How to Remediate**

Remove the unused staging model and their sources from the project.

---

## Models with Too Many Joins
Expand Down
6 changes: 6 additions & 0 deletions integration_tests/seeds/dag/dag_seeds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ seeds:
name: equality_fct_unused_sources
compare_model: ref('fct_unused_sources')

- name: test_fct_unused_staging_models
data_tests:
- dbt_utils.equality:
name: equality_fct_unused_staging_models
compare_model: ref('fct_unused_staging_models')

- name: test_fct_source_fanout
data_tests:
- dbt_utils.equality:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
parent
model.stage_model_4
model.stage_model_5
4 changes: 4 additions & 0 deletions models/marts/dag/dag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ models:
description: "This table shows each source with 0 children."
data_tests:
- is_empty
- name: fct_unused_staging_models
description: "This table shows each staging model with 0 children."
data_tests:
- is_empty
- name: fct_exposure_parents_materializations
description: "This table shows each direct parent of an exposure that is not materialized as a table or incremental."
data_tests:
Expand Down
23 changes: 23 additions & 0 deletions models/marts/dag/fct_unused_staging_models.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
-- this model finds cases where a source has no children

with source_relationships as (
select
*
from {{ ref('int_all_dag_relationships') }}
where parent_resource_type = 'model'
and parent_id like '%stg_%'
and not parent_is_excluded
and not child_is_excluded
),

final as (
select
parent
from source_relationships
group by 1
having max(distance) = 0
)

select * from final

{{ filter_exceptions() }}
Loading