From 8a13e3bd42ba92ff702cc69fcb0ff380620f6500 Mon Sep 17 00:00:00 2001 From: Nikita Peshkov Date: Mon, 9 Dec 2024 12:17:03 +0100 Subject: [PATCH 1/2] add artifacts_static_datasets var to upload_results --- README.md | 18 ++++++++++++++++++ macros/upload_results/upload_results.sql | 6 ++++++ 2 files changed, 24 insertions(+) diff --git a/README.md b/README.md index fa3f6ef0..d007e461 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,24 @@ Note that model materializations and `on_schema_change` configs are defined in t > Configurations made in your dbt_project.yml file will override any configurations in a package (either in the dbt_project.yml file of the package, or in config blocks). +### Artifact to upload + +By default, all non-execution artifacts are uploaded. Some users may be interested in choosing which datasets to upload. Thus, we allow setting `artifact_static_datasets` variable: + +```yml +vars: + artifacts_static_datasets: ['exposures', 'seeds', 'snapshots', 'invocations', 'sources', 'tests', 'models'] +``` + +> Note: execution-type arficats are always uploaded. + +It is possible to upload different sets of datasets depending on which dbt command is run. For example, upload all datasets on `dbt run`, upload only execution-type artifacts for other dbt invocations: + +```yml +vars: + artifacts_static_datasets: "{{ 'all' if invocation_args_dict.which == 'run' else [] }}" +``` + ### Environment Variables If the project is running in dbt Cloud, the following five columns () will be automatically populated in the fct_dbt__invocations model: diff --git a/macros/upload_results/upload_results.sql b/macros/upload_results/upload_results.sql index 114a667d..33c14e8f 100644 --- a/macros/upload_results/upload_results.sql +++ b/macros/upload_results/upload_results.sql @@ -5,6 +5,12 @@ {% if execute %} {% set datasets_to_load = ['exposures', 'seeds', 'snapshots', 'invocations', 'sources', 'tests', 'models'] %} + + {# When 'artifacts_static_datasets' var is provided, use it instead of default datasets #} + {% if var("artifacts_static_datasets", "all") != "all" %} + {% set datasets_to_load = fromjson(var("artifacts_static_datasets")) %} + {% endif %} + {% if results != [] %} {# When executing, and results are available, then upload the results #} {% set datasets_to_load = ['model_executions', 'seed_executions', 'test_executions', 'snapshot_executions'] + datasets_to_load %} From 80528ccf9b4e6dbc906b281e2bc0e8756e90aab5 Mon Sep 17 00:00:00 2001 From: Nikita Peshkov <90259794+npeshkov@users.noreply.github.com> Date: Mon, 9 Dec 2024 12:24:48 +0100 Subject: [PATCH 2/2] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d007e461..0b74600e 100644 --- a/README.md +++ b/README.md @@ -109,7 +109,7 @@ Note that model materializations and `on_schema_change` configs are defined in t > Configurations made in your dbt_project.yml file will override any configurations in a package (either in the dbt_project.yml file of the package, or in config blocks). -### Artifact to upload +### Selection of artifacts to upload By default, all non-execution artifacts are uploaded. Some users may be interested in choosing which datasets to upload. Thus, we allow setting `artifact_static_datasets` variable: