From e8f3e827919db0390eb102fb9e19cd8383668938 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Thu, 28 Jul 2022 13:46:58 +1200 Subject: [PATCH 1/8] First draft of package best practices --- README.md | 2 +- package-best-practices.md | 41 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 package-best-practices.md diff --git a/README.md b/README.md index f9e4c4c..6fa2984 100644 --- a/README.md +++ b/README.md @@ -14,4 +14,4 @@ Assorted constraints: ### Adding your package to hubcap Currently, only packages hosted on a GitHub repo are supported. -To add your package, open a PR that adds your repository to [hub.json](hub.json). +To add your package, open a PR that adds your repository to [hub.json](hub.json). A dbt Labs team member will review your PR and provide a cursory check of your new package against [best practices](package-best-practices.md). \ No newline at end of file diff --git a/package-best-practices.md b/package-best-practices.md new file mode 100644 index 0000000..3fea0e9 --- /dev/null +++ b/package-best-practices.md @@ -0,0 +1,41 @@ +# Best practices when submitting a package for public use + +Packages are a key part of the dbt experience. Practitioners and vendors alike can [contribute to the knowledge loop](https://github.com/dbt-labs/corp/blob/main/values.md#we-contribute-to-the-knowledge-loop) by enabling compelling, batteries-included use cases. + +There are hundreds of packages on the dbt Package Hub already, and over time we have identified a handful of common mistakes that new package authors make as they take a self-contained package and prepare it for publication. We have listed these below to help you and your users get the best experience. + +Although the dbt Labs team completes a cursory review of new packages before adding them to the Package Hub, ultimately you are responsible for your package's performance. + +To avoid ambiguity, the bullets below are pretty formal, but we are a friendly bunch! If you need help preparing a package for deployment to the dbt Package Hub, reach out in [#package-ecosystem](https://getdbt.slack.com/archives/CU4MRJ7QB/) in the [dbt Community Slack](https://getdbt.com/community). + +--- + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119). + +### First run experience +- Packages SHOULD contain a README file which explains how to get started with the package and customise its behaviour. + +### Customisability +- Packages MUST NOT hard-code table references, and MUST use `ref` or `source` instead. +- Packages MAY provide a mechanism (such as variables) to enable/disable certain subsets of functionality. +- Packages for data transformation: + - SHOULD provide a mechanism (such as variables) to customise the location of source tables. + - SHOULD NOT assume database/schema names in sources. + - MAY assume table names, particularly if the package was built to support tables created by a known tool. + +### Dependencies +- Packages requiring a minimum version of `dbt-core` MUST declare it in the `require-dbt-version` property of `dbt_project.yml`. +- Packages requiring a minimum version of `dbt-core` SHOULD declare compatibility with all subsequent minor and patch releases of that major version. + - For example, a package which depends on functionality added in dbt Core 1.2 SHOULD set its `require-dbt-version` property to `[">=1.2.0", "<2.0.0"]`. +Packages with a dependency on another package: + - MAY, if the dependent package is major-version-zero, pin to the current minor version only. + - SHOULD NOT pin to a patch version of the dependent package unless they are aware of an incompatibility between their package and the dependency. + - SHOULD reference the version in the dbt Package Hub when available, as opposed to a `git` installation. + +### Interoperability +- Packages MUST NOT override dbt Core behaviour in such a way as to impact other dbt resources (models, tests, etc) not provided by the package. +- Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. + - For example, packages SHOULD NOT provide a model simply called `users`. + +### Updates +- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). \ No newline at end of file From 111315e3d63f3355c0ef381c9d35ce04a1492d25 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Fri, 29 Jul 2022 14:31:40 +1200 Subject: [PATCH 2/8] Apply suggestions from code review Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- package-best-practices.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/package-best-practices.md b/package-best-practices.md index 3fea0e9..35ef799 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -14,6 +14,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ### First run experience - Packages SHOULD contain a README file which explains how to get started with the package and customise its behaviour. +- The documentation SHOULD indicate which data warehouses are expected to work with this package. ### Customisability - Packages MUST NOT hard-code table references, and MUST use `ref` or `source` instead. @@ -37,5 +38,6 @@ Packages with a dependency on another package: - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. -### Updates -- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). \ No newline at end of file +### Releases and updates +- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). +- Packages MUST create a git tag with the version number that follows the accepted version format. \ No newline at end of file From 9d560837c8a994c9490f357337563cb1ea2df370 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Fri, 29 Jul 2022 15:11:36 +1200 Subject: [PATCH 3/8] Tweaks from code review --- package-best-practices.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/package-best-practices.md b/package-best-practices.md index 35ef799..378a008 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -25,19 +25,22 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - MAY assume table names, particularly if the package was built to support tables created by a known tool. ### Dependencies +#### Dependencies on dbt Core - Packages requiring a minimum version of `dbt-core` MUST declare it in the `require-dbt-version` property of `dbt_project.yml`. -- Packages requiring a minimum version of `dbt-core` SHOULD declare compatibility with all subsequent minor and patch releases of that major version. +- Packages requiring a minimum version of `dbt-core` SHOULD allow all subsequent minor and patch releases of that major version. - For example, a package which depends on functionality added in dbt Core 1.2 SHOULD set its `require-dbt-version` property to `[">=1.2.0", "<2.0.0"]`. -Packages with a dependency on another package: - - MAY, if the dependent package is major-version-zero, pin to the current minor version only. - - SHOULD NOT pin to a patch version of the dependent package unless they are aware of an incompatibility between their package and the dependency. - - SHOULD reference the version in the dbt Package Hub when available, as opposed to a `git` installation. +#### Dependencies on other packages defined in `packages.yml`: +- Packages SHOULD import their dependencies from the dbt Package Hub when available, as opposed to a `git` installation. +- Packages SHOULD NOT pin to a patch version of their imported package unless they are aware of an incompatibility. +- Packages importing another package which has reached major version one or later SHOULD allow all subsequent minor and patch releases of that major version. +- Packages importing another package which is major version zero MAY pin to the current minor version only. ### Interoperability - Packages MUST NOT override dbt Core behaviour in such a way as to impact other dbt resources (models, tests, etc) not provided by the package. +- Packages MUST use the cross-database macros built into dbt Core where available, such as `{{ datediff }}` and `{{ type_string() }}`. - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. ### Releases and updates -- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). -- Packages MUST create a git tag with the version number that follows the accepted version format. \ No newline at end of file +- Packages' git tags MUST validate against the regex defined in [version.py](/hubcap/version.py). +- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). \ No newline at end of file From 4f996136808ef06d2971c04315a11f075cdd5bb5 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Tue, 2 Aug 2022 12:49:14 +1200 Subject: [PATCH 4/8] Update package-best-practices.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- package-best-practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package-best-practices.md b/package-best-practices.md index 378a008..95970a9 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -43,4 +43,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ### Releases and updates - Packages' git tags MUST validate against the regex defined in [version.py](/hubcap/version.py). -- Packages SHOULD follow the guidance of the [Semantic Versioning Specification](https://semver.org/). \ No newline at end of file +- Packages SHOULD follow the guidance of [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html). \ No newline at end of file From 378aba2b552204a0073d2496912e0ef5465638e2 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Tue, 2 Aug 2022 13:12:04 +1200 Subject: [PATCH 5/8] Code review --- package-best-practices.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/package-best-practices.md b/package-best-practices.md index 95970a9..6e8e3ac 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -31,13 +31,11 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - For example, a package which depends on functionality added in dbt Core 1.2 SHOULD set its `require-dbt-version` property to `[">=1.2.0", "<2.0.0"]`. #### Dependencies on other packages defined in `packages.yml`: - Packages SHOULD import their dependencies from the dbt Package Hub when available, as opposed to a `git` installation. -- Packages SHOULD NOT pin to a patch version of their imported package unless they are aware of an incompatibility. -- Packages importing another package which has reached major version one or later SHOULD allow all subsequent minor and patch releases of that major version. -- Packages importing another package which is major version zero MAY pin to the current minor version only. - +- Packages SHOULD specify the widest possible range of supported versions, to minimise issues in dependency resolution. + - In particular, packages SHOULD NOT pin to a patch version of their imported package unless they are aware of an incompatibility. ### Interoperability - Packages MUST NOT override dbt Core behaviour in such a way as to impact other dbt resources (models, tests, etc) not provided by the package. -- Packages MUST use the cross-database macros built into dbt Core where available, such as `{{ datediff }}` and `{{ type_string() }}`. +- Packages intended for use with multiple platforms MUST use the cross-database macros built into dbt Core where available, such as `{{ except() }}` and `{{ type_string() }}`. - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. From f74e5419b58fc8a1a299a45fff7f3e59055b67cc Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Wed, 3 Aug 2022 10:09:31 +1200 Subject: [PATCH 6/8] switch to SHOULD for xdb macros --- package-best-practices.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package-best-practices.md b/package-best-practices.md index 6e8e3ac..7f1bddc 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -35,7 +35,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - In particular, packages SHOULD NOT pin to a patch version of their imported package unless they are aware of an incompatibility. ### Interoperability - Packages MUST NOT override dbt Core behaviour in such a way as to impact other dbt resources (models, tests, etc) not provided by the package. -- Packages intended for use with multiple platforms MUST use the cross-database macros built into dbt Core where available, such as `{{ except() }}` and `{{ type_string() }}`. +- Packages SHOULD use the cross-database macros built into dbt Core where available, such as `{{ except() }}` and `{{ type_string() }}`. - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. From 8bc1d6b39a64f5ffe6d38e117b473ba0f9bc97f0 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Wed, 3 Aug 2022 16:41:42 +1200 Subject: [PATCH 7/8] please start at 1 --- package-best-practices.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package-best-practices.md b/package-best-practices.md index 7f1bddc..2b96c8c 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -39,6 +39,8 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. +### Releases and updates ### Releases and updates - Packages' git tags MUST validate against the regex defined in [version.py](/hubcap/version.py). -- Packages SHOULD follow the guidance of [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html). \ No newline at end of file +- Packages SHOULD follow the guidance of [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html). + - Note in particular the recommendation for production-ready packages to be version `1.0.0` or above \ No newline at end of file From f52677da31f162d1b9ab23863a5f78e4e1c4dbda Mon Sep 17 00:00:00 2001 From: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Date: Wed, 3 Aug 2022 08:57:50 -0600 Subject: [PATCH 8/8] Update package-best-practices.md --- package-best-practices.md | 1 - 1 file changed, 1 deletion(-) diff --git a/package-best-practices.md b/package-best-practices.md index 2b96c8c..bdeb6b6 100644 --- a/package-best-practices.md +++ b/package-best-practices.md @@ -39,7 +39,6 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - Packages SHOULD disambiguate their resource names to avoid clashes with nodes that are likely to already exist in a project. - For example, packages SHOULD NOT provide a model simply called `users`. -### Releases and updates ### Releases and updates - Packages' git tags MUST validate against the regex defined in [version.py](/hubcap/version.py). - Packages SHOULD follow the guidance of [Semantic Versioning 2.0.0](https://semver.org/spec/v2.0.0.html).