Deploy Prefect Server on K3s

[Jan-Willem]

Objective

Deploy the Prefect workflow from #13 on the cluster infrastructure provisioned by #27 to prove that an example pipeline can run on a deployment outside of developer sandboxes.

Prerequisite

• Develop an example Prefect workflow based on a typical pipeline #13
• k3s to facilitate MVP deployment #27

Definition of Done

• An interactive deployment of a prefect server dashboard exposed on port 4200 of a pod running on a node of the cluster created by k3s to facilitate MVP deployment #27

Artifacts

• Deployment configuration (e.g., prefect.yaml)
• Logs and outputs from remote workflow execution
• Notes on the design and implementation (RADPS/wiki/Kubernetes)

Success Criteria

• An example pipeline decomposition (RADPS/prefect_workflow/pipeline.py) is successfully executed on shared infrastructure.
• The Prefect server has the capability of running workflows submitted from multiple different requests.
• The Prefect dashboard and database services are resilient to lifecycle changes in the underlying pods.

Potential Challenges

Creating containers, resource definitions, network rules, ...

[amcnicho]

I've managed to create a basic Prefect deployment going on a test cluster (deployed on my workstation as part of #27). It's pure defaults at this point, except for a few tweaks necessary to get prefect-worker deployed.

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm install prefect-server prefect/prefect-server
helm install prefect-worker prefect/prefect-worker -f work_pool_values.yaml

At this point I'm trying to get the dashboard UI accessible from outside the host (i.e., from dispersed developer sandboxes, to see if we can use this as shared infrastructure to test the Prefect example workflows as we develop them). It seems like some changes beyond the standard kubectl port-forward svc/prefect-server 4200:4200 will be necessary.

After that, it'll be time to review prefect-worker/values.yaml and prefect-server/values.yaml to define how we want our deployments to look, and then start tracking these configurations in source control somewhere.

Note: so far cloudApiConfig.accountId seems to be a required setting even when using apiConfig.server or apiConfig.selfManaged.

[amcnicho]

The RADPS README now has instructions for starting up a basic test deployment of Prefect using the public helm charts. The input --values required to tweak the prefect-worker chart so it runs locally have been checked in under the directory charts/, which is probably where subsequent configuration changes should also be tracked for now.

[taktsutsumi]

Following the instructions in README I was able to install necessary packages to set up a local Kubernetes cluster environment on my intel Mac laptop. I found the instructions were clear and I did not run into any issues in testing a simple deployment of the Prefect pipeline script.