Develop an example Prefect workflow based on a typical pipeline

[taktsutsumi]

Objective

Develop a simple Prefect workflow based on the structure outlined in Figure 1 of the An example RADPS Workflow Decomposition. Summarize key learning in a presentation.

Prerequisite

• Select and complete Prefect Tutorial  #11

Requirements

A Prefect workflow replicating the structure of Figure 1 in the An example RADPS Workflow Decomposition .

• Tasks can be no-ops or simple sleep operations.
• The workflow should be run locally to verify execution.
• Include at least one conditional loop to demonstrate dynamic behavior.

Definition of Done

• A Prefect workflow mimicking the structure in Figure 1 is created.
• Tasks in the workflow use placeholders (e.g., no-op or sleep tasks) to demonstrate flow structure.
• At least one conditional loop is added to the workflow to illustrate dynamic behavior.
• The workflow is successfully executed in a local environment.
• The workflow includes basic reliability and fault tolerance features, such as retries and failure recovery.
• Notes on the design and implementation.

Key Decision Points

1. Figure 1 Workflow Interpretation:
   • Agree on the structure and task definitions for the workflow based on Figure 1.
2. Conditional Task Design:
   • Decide how and where conditional tasks or task branching should be integrated into the workflow.

Artifacts

• A functional Prefect workflow script
• Logs and outputs from local workflow execution
• Notes on the design and implementation

Success Criteria

A working Prefect workflow, including conditional loops and failure handling, is successfully run locally.
The team is equipped to design and implement more complex workflows in the future.

Potential Challenges

• Correctly interpreting and implementing Figure 1's workflow structure.
• Troubleshooting environment or execution issues during local workflow testing.
• Implementing conditional loops correctly and ensuring their functionality in the workflow.
• Reconciling the data structures used as input/output of the tasks comprising the workflow sequence.