📖 Airflow Deployments Trial

[jhpyke]

User Story

As a maintainer of CaDeT Deployments via Airflow
I need to run a trial of deploying via Airflow
So that I can be confident it is a suitable solution for our deployments.

Value / Purpose

We're collecting signups for deployments via Airflow, with signups closing next Wednesday CoP. As those signups come in, we should be creating deployments for them so that they can see that Airflow can handle their deployments.

Useful Contacts

@jhpyke

User Types

No response

Hypothesis

No response

Proposal

We should support two broad categories of deployment:
Dual Running - Avoid interfering with existing prod runs:

• Deploys against Sandpit
• Runs on same schedule as currently, but 12 hours delayed.

Solo Running:

• Deploys Against Prod
• Runs on the schedule of the existing github action
• Existing github action should be moved to workflow_dispatch.

Additional Information

Workflows should be spun up with a particular attention to the --target - assume users don't want prod unless they either specify they do, or specify solo running.

Definition of Done

• DAGs exist for all participating workflows.
• Dual Running DAGs are able to deploy on their shifted schedules
• DAGs are monitored for errors and those are resolved by the end of the test period.