Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] customize pipeline with hooks #63

Open
rudolfix opened this issue Sep 13, 2022 · 0 comments
Open

[core] customize pipeline with hooks #63

rudolfix opened this issue Sep 13, 2022 · 0 comments
Assignees

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Sep 13, 2022

action to be triggered by certain events in the pipeline can be delegated by the pipeline developer. examples of events and actions:

  • load job permanently failed. user code gets error message, can inspect the affected file and make decision to proceed or abort the whole load
  • new table or new column was inferred. user may change the data type, write disposition or ignore the change (which will drop the data)
  • variant column is being created due to inconsistent types. options as above
  • value cannot be coerced into the column type. user can provide custom casting function (ie. for text->datetime conversions)
  • relational json normalizer recurs into nested table and calls a hook. based on table name, parent table name, nesting level etc. you can decide to create complex type that will serialize the nested elements into json or proceed with recursion or drop the data
  • when writing text value into the loader file we exceed VARCHAR(MAX) length: user can cut the string or raise error (or ignore and try his luck)

[we need more events]

we need a clean way to define and add hook functions. investigate if https://github.com/getslash/gossip fits.

@rudolfix rudolfix self-assigned this Sep 13, 2022
@rudolfix rudolfix moved this from Todo to In Progress in dlt core library Jul 30, 2023
@rudolfix rudolfix moved this from In Progress to Todo in dlt core library Jul 30, 2023
@rudolfix rudolfix assigned sh-rp and unassigned rudolfix Dec 18, 2023
@rudolfix rudolfix moved this from Todo to Planned in dlt core library Dec 18, 2023
@sh-rp sh-rp mentioned this issue Dec 18, 2023
3 tasks
@sh-rp sh-rp moved this from Planned to In Progress in dlt core library Jan 8, 2024
@rudolfix rudolfix moved this from In Progress to Planned in dlt core library Feb 6, 2024
@sh-rp sh-rp moved this from Planned to Todo in dlt core library Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

2 participants