Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when yielding data from a resource, mark given value to be normalized #153

Open
rudolfix opened this issue Feb 23, 2023 · 0 comments
Open

Comments

@rudolfix
Copy link
Collaborator

Goal
We want to mark certain string values to be normalized according to naming convention used later in the pipeline. Compare such feature to dlt.with_table where we mark whole dictionary to be routed to table with given type.

Motivation
Quite often some lineage metadata is loaded with the data itself. For example in case of pipedrive we load a table with definitions of user defined fields. Those fields are present in dlt schema and their names are normalized like any other column names.

Tasks

    • add normalize function in dlt.mark module. that will wrap a str in a string like object that will be serialized in typed_dump as a special type (like for example Decimal or datetime)
    • when deserialized in normalizer, a normalize_identifier method from the current schema context will be applied to the value
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant