Versioning and extensions

[roll]

• fixes Data Package versioning and profiles simplification datapackage#896
• fixes Profile is a url rather than string datapackage#689
• fixes Adding a version to profile attribute (or doing this via a "spec" namespace) datapackage#444

---

Overview

It's the beginning of the pull request that might bring versioning to Data Package specification descriptors as well as simplify and fix some other parts of the standard.

Currently, it's been created to get feedback on the new $schema property that allows versioning and hopefully will allow us to employ native JSON Schema for handling basically all the metadata aspects in the standard (PR needs to be extended to get there).

One of examples of how we can benefit from closer JSON Schema integration, by using $schema we get free visual descriptor validation in many modern text editors:

---

Deployment preview:

• $schema
• Profile
• Extensions

[cloudflare-workers-and-pages]

Deploying datapackage with    Cloudflare Pages

Latest commit:	913e41a
Status:	✅  Deploy successful!
Preview URL:	https://1432657e.datapackage.pages.dev
Branch Preview URL:	https://444-versioning.datapackage.pages.dev

View logs

[roll]

Hi @peterdesmet,

I was going back and forth on this, and in the end, I highly simplified the approach to the state similar to what we have in v1 but improving explanations and definitions. The main changes are:

• requiring profiles to be idempotent in relation to their URL (I think on the implementation level, we don't really care about data package version; what we really care about is ensuring that a descriptor actually valid against its profile as it's self-containable; idempotency solves this problem)
• requiring profiles to include default validation rules

[roll]

For example, if profiles are idempotent, we can safely release e.g. https://datapackage.org/profiles/3.0/datapackage.json that have breaking change but all the existent data packages won't be affected because a data publisher needs explicitly update a profile version to migrate.

---

At the same time, idempotency might be too strict as it prevents implicit bug-fixing in the profiles and providing new, non-breaking features. So another option would be switching from idempotent to non-breaking and having e.g. major-version based versioning like https://datapackage.org/profiles/v3/datapackage.json

[peterdesmet]

I think I prefer idempotent (immutable) to non-breaking: it makes the text pretty clear and indicates to extension authors that you should be careful not to release versions that are inherently broken. 😅 That's why we used a release candidate for Camtrap DP, making it clear that any data artefact using that is experimental.

[roll]

Thanks @peterdesmet! I've updated it based on your comments.

I think immutability is definitely the most straightforward way, as it enables worry-free caching and, in general, a really simple concept on both sides (standard/extensions -> implementations)

[pschumm]

I believe immutable is indeed what we want here. Unless I'm misunderstanding something, I don't believe the term idempotent is appropriate in this case (that has a very specific mathematical definition, and I don't see how it applies here).

[roll]

ACCEPTED by WG (6/9)

[khughitt]

I love this. Awesome work. 👌