Merge pull request #48 from fivetran/MagicBot/dbt-utils-cross-db-migr…

…ation

Updates for dbt-utils to dbt-core cross-db macro migration

CHANGELOG.md

@@ -1,3 +1,42 @@
+# dbt_stripe v0.8.0
+
+## 🚨 Breaking Changes 🚨:
+[PR #48](https://github.com/fivetran/dbt_stripe/pull/48) includes the following breaking changes:
+- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically `{{ dbt_utils.<macro> }}` have been updated to `{{ dbt.<macro> }}` for the below macros:
+    - `any_value`
+    - `bool_or`
+    - `cast_bool_to_text`
+    - `concat`
+    - `date_trunc`
+    - `dateadd`
+    - `datediff`
+    - `escape_single_quotes`
+    - `except`
+    - `hash`
+    - `intersect`
+    - `last_day`
+    - `length`
+    - `listagg`
+    - `position`
+    - `replace`
+    - `right`
+    - `safe_cast`
+    - `split_part`
+    - `string_literal`
+    - `type_bigint`
+    - `type_float`
+    - `type_int`
+    - `type_numeric`
+    - `type_string`
+    - `type_timestamp`
+    - `array_append`
+    - `array_concat`
+    - `array_construct`
+- For `current_timestamp` and `current_timestamp_in_utc` macros, the dispatch AND the macro names have been updated to the below, respectively:
+    - `dbt.current_timestamp_backcompat`
+    - `dbt.current_timestamp_in_utc_backcompat`
+- Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
+- Updated README to include instructions on how to use metadata variable in cases of dictionary arguments. ([#51](https://github.com/fivetran/dbt_stripe/pull/51))
 # dbt_stripe v0.7.4
 ## 🎉 Feature Updates
 - Included the `subscription_item_id` field within the `stripe__invoice_line_items` model. ([#50](https://github.com/fivetran/dbt_stripe/pull/50))

README.md

@@ -3,7 +3,7 @@
         href="https://github.com/fivetran/dbt_stripe/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
     <a alt="dbt-core">
-        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.0.0_,<2.0.0-orange.svg" /></a>
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_,<2.0.0-orange.svg" /></a>
     <a alt="Maintained?">
         <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
     <a alt="PRs">
@@ -55,7 +55,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=0.7.0", "<0.8.0"]
+    version: [">=0.8.0", "<0.9.0"]
 
 ```
 ## Step 3: Define database and schema variables
@@ -109,21 +109,48 @@ vars:
         using_invoice_line_sub_filter: false # Default = true
 ```
 
+
 ### Pivoting out Metadata Properties
-By default, this package selects the `metadata` JSON field within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. However, you likely have properties within the `metadata` JSON field you would like to pivot out and include in the respective downstream staging model.
+Oftentimes you may have custom fields within your source tables that is stored as a JSON object that you wish to pass through. By leveraging the `metadata` variable, this package pivot out fields into their own columns. The metadata variables accept dictionaries in addition to strings.
+
+Additionally, if you happen to be using a reserved word as a metadata field, any otherwise incompatible name, or just wish to rename your field, Below are examples of how you would add the respective fields.
+
+The `metadata` JSON field is present within the `customer`, `charge`, `invoice`, `payment_intent`, `payment_method`, `payout`, `plan`, `refund`, and `subscription` source tables. To pivot these fields out and include in the respective downstream staging model, add the respective variable(s) to your root `dbt_project.yml` file like below.
+
+```yml
+vars: 
+  stripe__charge_metadata:
+    - name: metadata_field_1
+  stripe__invoice_metadata: 
+    - name: metadata_field_2
+  stripe__payment_intent_metadata:
+    - name: incompatible.field
+      alias: rename_incompatible_field
+  stripe__payment_method_metadata:
+    - name: field_is_reserved_word
+      alias: field_is_reserved_word_xyz
+  stripe__payout_metadata:
+    - name: 123
+      alias: one_two_three
+  stripe__plan_metadata:
+    - name: rename
+    - alias: renamed_field
+  stripe__refund_metadata:
+    - name: metadata_field_3
+    - name: metadata_field_4
+  stripe__subscription_metadata:
+    - name: metadata_field_5
+  stripe__customer_metadata:
+    - name: metadata_field_6
+
+```
+
+Alternatively, if you only have strings in your JSON object, the metadata variable accepts the following configuration as well. 
+>**Note**: `stripe__plan_metadata` is only shown below, but the format will work for all metadata variables. 
 
-If there are properties in the `metadata` JSON field that you'd like to pivot out into columns, add the respective variable(s) to your root `dbt_project.yml` file:
 ```yml
 vars:
-    stripe__charge_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__invoice_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payment_intent_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payment_method_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__payout_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
     stripe__plan_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__refund_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
-    stripe__customer_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
 ```
 
 ### Change the build schema
@@ -160,13 +187,16 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/stripe_source
-      version: [">=0.7.0", "<0.8.0"]
+      version: [">=0.8.0", "<0.9.0"]
 
     - package: fivetran/fivetran_utils
-      version: [">=0.3.0", "<0.4.0"]
+      version: [">=0.4.0", "<0.5.0"]
 
     - package: dbt-labs/dbt_utils
-      version: [">=0.8.0", "<0.9.0"]
+      version: [">=1.0.0", "<2.0.0"]
+
+    - package: dbt-labs/spark_utils
+      version: [">=0.3.0", "<0.4.0"]
 ```
 # 🙌 How is this package maintained and can I contribute?
 ## Package Maintenance

dbt_project.yml

@@ -1,8 +1,8 @@
 config-version: 2
 name: 'stripe'
-version: '0.7.4'
 
-require-dbt-version: [">=1.0.0", "<2.0.0"]
+version: '0.8.0'
+require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   stripe:
     +schema: stripe
@@ -27,4 +27,4 @@ vars:
     subscription: "{{ ref('stg_stripe__subscription') }}"
     using_invoices: true
     using_payment_method: true
-    using_subscriptions: true
+    using_subscriptions: true

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: stripe_integrations_tests
+      schema: stripe_integrations_tests_11
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: stripe_integrations_tests
+      schema: stripe_integrations_tests_11
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: stripe_integrations_tests
+      schema: stripe_integrations_tests_11
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: stripe_integrations_tests
+      schema: stripe_integrations_tests_11
       threads: 8
     databricks:
       catalog: null
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: stripe_integrations_tests
+      schema: stripe_integrations_tests_11
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,12 +1,12 @@
 config-version: 2
 
 name: 'stripe_integration_tests'
-version: '0.7.4'
+version: '0.8.0'
 
 profile: 'integration_tests'
 
 vars:
-  stripe_schema: stripe_integrations_tests
+  stripe_schema: stripe_integrations_tests_11
   stripe_source:
     stripe_group_identifier:                  "group_data"
     stripe_balance_transaction_identifier:    "balance_transaction_data"

integration_tests/requirements.txt

@@ -1,7 +1,7 @@
-dbt-snowflake>=1.0.0,<1.3.0
-dbt-bigquery>=1.0.0,<1.3.0
-dbt-redshift>=1.0.0,<1.3.0
-dbt-postgres>=1.0.0,<1.3.0
-dbt-spark>=1.0.0,<1.3.0
-dbt-spark[PyHive]>=1.0.0,<1.3.0
-dbt-databricks>=1.0.0,<1.3.0
+dbt-snowflake>=1.3.0,<2.0.0
+dbt-bigquery>=1.3.0,<2.0.0
+dbt-redshift>=1.3.0,<2.0.0
+dbt-postgres>=1.3.0,<2.0.0
+dbt-spark>=1.3.0,<2.0.0
+dbt-spark[PyHive]>=1.3.0,<2.0.0
+dbt-databricks>=1.3.0,<2.0.0

macros/date_timezone.sql

@@ -25,7 +25,7 @@ date(
 
 {% endset %}
 
-{{ dbt_utils.date_trunc('day',converted_date) }}
+{{ dbt.date_trunc('day',converted_date) }}
 
 {%- endmacro %}
 
@@ -42,7 +42,7 @@ date(
 
 {% endset %}
 
-{{ dbt_utils.date_trunc('day',converted_date) }}
+{{ dbt.date_trunc('day',converted_date) }}
 
 {%- endmacro %}
 
@@ -59,7 +59,7 @@ date(
 
 {% endset %}
 
-{{ dbt_utils.date_trunc('day',converted_date) }}
+{{ dbt.date_trunc('day',converted_date) }}
 
 {%- endmacro %}
 
@@ -75,6 +75,6 @@ date(
 
 {% endset %}
 
-{{ dbt_utils.date_trunc('day',converted_date) }}
+{{ dbt.date_trunc('day',converted_date) }}
 
 {%- endmacro %}

models/stripe__balance_transactions.sql

@@ -71,7 +71,7 @@ select
     balance_transaction.description,
     case when balance_transaction.type = 'charge' then charge.amount end as customer_facing_amount, 
     case when balance_transaction.type = 'charge' then charge.currency end as customer_facing_currency,
-    {{ dbt_utils.dateadd('day', 1, 'balance_transaction.available_on') }} as effective_at,
+    {{ dbt.dateadd('day', 1, 'balance_transaction.available_on') }} as effective_at,
     coalesce(charge.customer_id, refund_charge.customer_id) as customer_id,
     charge.receipt_email,
     customer.description as customer_description, 

models/stripe__customer_overview.sql

@@ -36,31 +36,31 @@ with balance_transaction_joined as (
         then 1
         else 0 
             end) as total_refund_count,   
-      sum(case when type in ('charge', 'payment') and {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when type in ('charge', 'payment') and {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then amount 
         else 0 
             end) as sales_this_month,
-      sum(case when type in ('payment_refund', 'refund') and {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when type in ('payment_refund', 'refund') and {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then amount 
         else 0 
             end) as refunds_this_month,
-      sum(case when {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then amount 
         else 0 
             end) as gross_transaction_amount_this_month,
-      sum(case when {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then fee 
         else 0 
             end) as fees_this_month,
-      sum(case when {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then net 
         else 0 
             end) as net_transaction_amount_this_month,
-      sum(case when type in ('charge', 'payment') and {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when type in ('charge', 'payment') and {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then 1 
         else 0 
             end) as sales_count_this_month,
-      sum(case when type in ('payment_refund', 'refund') and {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when type in ('payment_refund', 'refund') and {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then 1 
         else 0 
             end) as refund_count_this_month,
@@ -82,11 +82,11 @@ with balance_transaction_joined as (
       customer_id,
       count(*) as total_failed_charge_count,
       sum(amount) as total_failed_charge_amount,
-      sum(case when {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then 1
         else 0 
             end) as failed_charge_count_this_month,
-      sum(case when {{ dbt_utils.date_trunc('month', date_timezone('created_at')) }} = {{ dbt_utils.date_trunc('month', date_timezone(dbt_utils.current_timestamp())) }}
+      sum(case when {{ dbt.date_trunc('month', date_timezone('created_at')) }} = {{ dbt.date_trunc('month', date_timezone(dbt.current_timestamp_backcompat())) }}
         then amount
         else 0 
             end) as failed_charge_amount_this_month

models/stripe__monthly_overview.sql

@@ -6,7 +6,7 @@ with daily_overview as (
 )
 
 select
-    {{ dbt_utils.date_trunc('month', 'date')}} as month,
+    {{ dbt.date_trunc('month', 'date')}} as month,
     sum(total_sales) as total_sales,
     sum(total_refunds) as total_refunds,
     sum(total_adjustments) as total_adjustments,

models/stripe__quarterly_overview.sql

@@ -6,7 +6,7 @@ with daily_overview as (
 )
 
 select
-    {{ dbt_utils.date_trunc('quarter', 'date')}} as quarter,
+    {{ dbt.date_trunc('quarter', 'date')}} as quarter,
     sum(total_sales) as total_sales,
     sum(total_refunds) as total_refunds,
     sum(total_adjustments) as total_adjustments,

models/stripe__weekly_overview.sql

@@ -6,7 +6,7 @@ with daily_overview as (
 )
 
 select
-    {{ dbt_utils.date_trunc('week', 'date')}} as week,
+    {{ dbt.date_trunc('week', 'date')}} as week,
     sum(total_sales) as total_sales,
     sum(total_refunds) as total_refunds,
     sum(total_adjustments) as total_adjustments,

packages.yml

@@ -1,3 +1,3 @@
 packages:
-  - package: fivetran/stripe_source
-    version: [">=0.7.0", "<0.8.0"]
+- package: fivetran/stripe_source
+  version: [">=0.8.0", "<0.9.0"]