[RFC] A multiplexfeature for Workflows

[zeddo123]

When defining a workflow, in some cases the output received from a previous step needs to be iterated through and applied to the same processing/function.

let's say we have wf1, with a first step that returns a list of bboxes [[0, 0, 10, 10], [23, 23, 45, 33], ...]. Now, in our 2nd step we're running an inference on a model for each bbox supplied by the first step. This operation would take too long if the 2nd step is only implemented as a simple for loop (especially when running the pipeline needs to happen as quick as possible).

My proposition would be for octopipes to provide a "multiplex" function similar to the "add" function that takes care of running simultaneously the inputs of a previous step on the current step and then collecting the results when propagating them to the next steps (if present).

Here's an example of such a feature

wf = Workflow('wf1').add(load_data).multiplex(run_model, workers=3).add(post_process)