You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
A few issues in current dlt deploy after #356 got merged remain and should be fixed. The most important is to add an CLI option to let the user choose a format in which s/he wants to add the secret values to Airflow. Currently we always show them as environment variables.
Tasks
allow to pick the format of the secrets/config. we currently always generate the environment variables. We want to add the options to generate a toml snippet that you can paste into dlt_secrets_toml Ariflow Variable. if user picks the toml option we display a toml document and instructions what to do with it.
Backlog Tasks
allow vault secrets format - we generate toml fragments compatible with the Google Secrets or Airflow config provider with the instructions where to place them.
we must set the schedule in @dag of the dag script, this of course means that we need to start parsing the template file with AST. (I do want no jinjas as long as possible)
we should probably use subparser for each deployment type so they do not overlap
Implementation Details
BaseTomlProvider has now a nice method to write config values back to the document (set_value). To generate secrets.toml to display just write all the config data back to an empty instance. You'll probably need to create MemoryTomlProvider because base class is abstract.
vault format will be soon documented. we can start with secrets.toml format
we could put everything in default_args so rewriting the AST is easier
Look at AirflowSecretsTomlProvider and its base class to see how dlt_secrets_toml and vault toml is organized. the tests will also help you
The text was updated successfully, but these errors were encountered:
Background
A few issues in current
dlt deploy
after #356 got merged remain and should be fixed. The most important is to add an CLI option to let the user choose a format in which s/he wants to add the secret values to Airflow. Currently we always show them as environment variables.Tasks
toml
snippet that you can paste intodlt_secrets_toml
Ariflow Variable. if user picks thetoml
option we display a toml document and instructions what to do with it.Backlog Tasks
vault
secrets format - we generate toml fragments compatible with the Google Secrets or Airflow config provider with the instructions where to place them.Implementation Details
BaseTomlProvider
has now a nice method to write config values back to the document (set_value). To generatesecrets.toml
to display just write all the config data back to an empty instance. You'll probably need to createMemoryTomlProvider
because base class is abstract.secrets.toml
formatdefault_args
so rewriting the AST is easierAirflowSecretsTomlProvider
and its base class to see howdlt_secrets_toml
and vault toml is organized. the tests will also help youThe text was updated successfully, but these errors were encountered: