Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve the CLI for dlt deploy #368

Open
rudolfix opened this issue May 29, 2023 · 0 comments
Open

improve the CLI for dlt deploy #368

rudolfix opened this issue May 29, 2023 · 0 comments
Labels
good first issue Good for newcomers

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented May 29, 2023

Background
A few issues in current dlt deploy after #356 got merged remain and should be fixed. The most important is to add an CLI option to let the user choose a format in which s/he wants to add the secret values to Airflow. Currently we always show them as environment variables.

Tasks

    • allow to pick the format of the secrets/config. we currently always generate the environment variables. We want to add the options to generate a toml snippet that you can paste into dlt_secrets_toml Ariflow Variable. if user picks the toml option we display a toml document and instructions what to do with it.

Backlog Tasks

    • allow vault secrets format - we generate toml fragments compatible with the Google Secrets or Airflow config provider with the instructions where to place them.
    • we must set the schedule in @dag of the dag script, this of course means that we need to start parsing the template file with AST. (I do want no jinjas as long as possible)
    • we should probably use subparser for each deployment type so they do not overlap

Implementation Details

  1. BaseTomlProvider has now a nice method to write config values back to the document (set_value). To generate secrets.toml to display just write all the config data back to an empty instance. You'll probably need to create MemoryTomlProvider because base class is abstract.
  2. vault format will be soon documented. we can start with secrets.toml format
  3. we could put everything in default_args so rewriting the AST is easier
  4. Look at AirflowSecretsTomlProvider and its base class to see how dlt_secrets_toml and vault toml is organized. the tests will also help you
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
Status: Todo
Development

No branches or pull requests

2 participants