Local validation of a team pipeline

[cmaumet]

Note: This issue is a first version that might be reworked in the future.

Goal

The goal is to implement (and document) how to run the pipeline testing locally so that we can validate that a reproduction (e.g. provided in a PR by an external contributor) is successful. A successful reproduction is a piece of code associated with one of the NARPS team that we can run and for which we re-obtain the same results (i.e. correlation > 0.98) as the one published by the team on NeuroVault.

Why is this useful?

It will be useful to be able to run the validation code locally for two reason:

1. This can be used to decide whether we can merge a PR while we are waiting to find a solution to have this directly in the CI
2. This can also be useful for contributors to try and validate their own code

Timeline

We'd like a first version for the start of the OHBM Brainhack 2023 - July 19-21, 2023.

More specifically

Open questions:

[cmaumet]

I think this comment #46 (comment) by @elodiegermani is related :D

[cmaumet]

Discussed with @bclenet on Jan 9, this is documented in testing.md. Creating a new function/module pipeline.get_correlation that can be run once the pipeline has been implemented to see what the correlation values are.