You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
We want to be able to add additional metadata to dlt objects. The first use is to label schemas so we know if they were created by pipeline, source or by user.
The second use is to label tables and columns pre or during extraction so those labels are available to downstream system that do transformations of dlt generated data.
Labelling of classes should happen via common mixin from which dlt entities should derive. Mixin should allow to accept a json serializable dictionary and expose it as meta property.
Labelling of schema entities should happen via base TypedDict
rudolfix
changed the title
[WIP] support custom metadata in dlt entities
[WIP] support custom metadata in dlt entities and table and column level labelling
Nov 23, 2023
Background
We want to be able to add additional metadata to dlt objects. The first use is to label schemas so we know if they were created by pipeline, source or by user.
The second use is to label tables and columns pre or during extraction so those labels are available to downstream system that do transformations of dlt generated data.
Labelling of classes should happen via common mixin from which dlt entities should derive. Mixin should allow to accept a json serializable dictionary and expose it as
meta
property.Labelling of schema entities should happen via base
TypedDict
Which entities
Tasks
meta
argument in Pipeline, DltSource, DltResource, Schema and DestinationClientConfiguration (and destination factory when implemented)meta
argument indlt.source
anddlt.resource
decoratorsdlt.mark.label
that allows to add table and column level labels to a data item during extraction.The text was updated successfully, but these errors were encountered: