Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] support custom metadata in dlt entities and table and column level labelling #761

Open
rudolfix opened this issue Nov 12, 2023 · 0 comments

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Nov 12, 2023

Background
We want to be able to add additional metadata to dlt objects. The first use is to label schemas so we know if they were created by pipeline, source or by user.

The second use is to label tables and columns pre or during extraction so those labels are available to downstream system that do transformations of dlt generated data.

Labelling of classes should happen via common mixin from which dlt entities should derive. Mixin should allow to accept a json serializable dictionary and expose it as meta property.

Labelling of schema entities should happen via base TypedDict

Which entities

  • Pipeline
  • destination client configuration (via destination factory)
  • DltSource
  • DltResource
  • Schema
  • Tables
  • Columns

Tasks

    • implement Meta mixin
    • accept meta argument in Pipeline, DltSource, DltResource, Schema and DestinationClientConfiguration (and destination factory when implemented)
    • accept meta argument in dlt.source and dlt.resource decorators
    • column and table TypeDicts allow for a "meta" dict with the intention for custom column and table labelling.
    • implement dlt.mark.label that allows to add table and column level labels to a data item during extraction.
@rudolfix rudolfix added the devel label Nov 21, 2023
@rudolfix rudolfix changed the title [WIP] support custom metadata in dlt entities [WIP] support custom metadata in dlt entities and table and column level labelling Nov 23, 2023
@rudolfix rudolfix removed the devel label Dec 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant