Auto-execute pipeline on new imported data (e.g. SIMS)

[michaelkain]

Hi @jcomedouteau,
I hope you are doing fine.
This should be a very interesting extension to work on.
The idea is to configure a pipeline-run in VIP with a specific pipeline,
once the import has been done. So this might be done with a daily
job, that runs and initiates the necessary executions depending on
some schema. With kind regards, Michael
PS: a simple start might be: "on every new exam" run "X"

[michaelkain]

Hi @jcomedouteau, I think this might become a really interesting extension for entire Shanoir+VIP.
The idea to trigger a specific pipeline, based on some criteria, run it and get back the result, will be
an important baseline for future studies.

[jcomedouteau]

Hi Michael,
This is an idea I discussed with benjamin too, it would be a great forward step for shanoir!
The question is how we implement it:

• Integrated in a study card ? (instead of renaming/adding metadata, start a process)
• Add a new "automatic action" in study, with a possibility to filter / manage new datas.

This can be quite simple at the beginning and become something quite huge, we need to think of it to be able to evolve.

[jcomedouteau]

Specifications for pipeline automatic execution

FRONT

STUDY

Add a selector on pipelines. (if no available pipelines -> Either do not display the tab or disable it)

Add a new tab in study detail
-> Configure a new "automatic execution".
-> Add filter on center
-> Add filter on acquisition
-> Add filter on subject (example -> subject name has to contain ES for spanish subjects only)
-> Add filter on examination name / date (example all exams with name starting with FLI)
-> Add filter on acquisition name
-> Add filter on dataset name (consider the pipeline only for exams with a dataset containing "Crane sans IV"

On every level, add the possibility to take all by default.
(Display an example of current datasets matching these ?)

Then display the pipeline configuration with arguments as configured in the "execute pipeline" page.
-> Name automatically generated
-> For non datasets argument, propose an empty textbox and/or boolean selectors
-> For datasets arguments, a textbox with a regex field to choose adapted datasets (if *, we send all datasets ????????)

IMPORT

During an import, during clinical context, add a "tab" (vertical), if the import can match a pipeline execution (warning, this will trigger an execution of type XXX (show some detail ?), and add a checkbox to be able to uncheck it if necessary.

Create a new structure

ExecutionPlanification {
// VIP pipeline
Pipeline pipeline; (to be completed in shanoir too ? Or directly load it from VIP ?)

// default pipieline arguments ?

// list of String filters
String centerFilter;
...
String datasetFilter;

}

An associated service/impl

ExecutionService{
// Create a new executionPlanification
createExecutionPlanification(EP)

// Check if an exam structure respects an planification filter
checkExecutionPlanification(EP, Examination)

// Create an execution linked to a planification
createPlanifiedExecution(EP, Exam); 

}

At the end of import (asynchroneously i guess), check every potentiel execution planification, and create executions for those that matches.

[michaelkain]

Hi @DuckflipXYZ, please provide me a short todo-list, that you see behind this issue in relation with the PR already existing and how much time it might require you please. Thank you!

[DuckflipXYZ]

Hi @michaelkain ! Jean-Come almost did everything ! What's missing is the exec logic (Should we use CRON or deal with the exec directly after import ? If CRON, what freq ? Or maybe the freq is a parameter ?

I didn't test Jean-Come PR yet, so there is a debug/test stage too. I think a work time of 3 days until a week seems relevant.