08MQ reproduction (#121)

* [BUG] inside unit_tests workflow

* [08MQ] Preprocessing script in progress

* Working on preprocessing

* Codespell

* Working on pipeline 08MQ

* Towards a running version of the pipeline

* Conforming to Pipeline class + init test module

* Towards a running version of the pipeline

* Towards a running version of the pipeline

* Towards a running version of the pipeline

* [TEST] runner test not using 08MQ as example anymore

* Issue with SelectFiles' base_directory

* Towards a running version of the pipeline

* [TEST] stop using 08MQ in tests

* 08MQ preprocessing instantiable

* Compute confounds node

* Group analysis workflow bug

* Group analysis workflow bug

* Group analysis workflow bug

* Issue with Registration and Threshold

* Bug with shrink factors

* Bug with shrink factors

* Bug with shrink factors

* ANTs registration [skip ci]

* Testing parameters for antsRegistration [skip ci]

* Inverse coregistration transform [skip ci]

* Inverse coregistration transform [skip ci]

* Compute confounds parameterization [skip ci]

* ANTs composite transform [skip ci]

* Trying ANTs' ApplyTransforms

* Preprocessing outputs [skip ci]

* Preprocessing outputs [skip ci]

* Add smoothing [skip ci]

* Motion correction outputs [skip ci]

* Parse segmentation maps [skip ci]

* Use partial volume files from segmentation [skip ci]

* Use partial volume files from segmentation [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* ANT's apply transform bug [skip ci]

* ANT's apply transform bug [skip ci]

* ANT's apply transform bug [skip ci]

* ANT's apply transform bug [skip ci]

* ANT's apply transform bug [skip ci]

* First try of run level analysis [skip ci]

* First try of run level analysis [skip ci]

* First try of subject level analysis [skip ci]

* First try of subject level analysis [skip ci]

* FSL's Merge disambiguataion [skip ci]

* Use MNI masks [skip ci]

* Preprocessing sieable files removing

* Bug with contrasts Node naming

* Base directories

* Remove alignment_func_to_mni's output [skip ci]

* Autoremove intermediate results [skip ci]

* Autoremove intermediate results [skip ci]

* Revert "Remove alignment_func_to_mni's output [skip ci]"

This reverts commit 4dad129.

* Bug after revert [skip ci]

* Bug after revert [skip ci]

* [TEST] tests updates

* Bug with big files removal

* Deletinglarge files after datasink

* [BUG] group level workflow connections [skip ci]

* [BUG] group level [skip ci]

* [REFAC] equalRange > equal_range [skip ci]

* Regressors function + group output [skip ci]

* Typo in regressors [skip ci]

* Issue with list of  regressors [skip ci]

* Issue with list of  regressors [skip ci]

* Issue with list of  regressors [skip ci]

* [TEST] adding unit tests to 08MQ [skip ci]

* [TEST] typo [skip ci]

* [TEST] static methods testing [skip ci]

* [TEST] adding unit tests for 08MQ

* [CODESPELL] typos

* [BUG] inversion between groups [skip ci]

* [REFAC] Using narps_open.core functions

* [REFAC] using narps_open.core functions [TEST] events data in a file

* [BUG] remove_file input naming [skip ci]

* [BUG] remove sub- from participants id [skip ci]

* [BUG] typo in get_two_sample_t_test_regressors inputs [skip ci]

* Compute brightness threshold for SUSAN [skip ci]

* Use ANTs ApplyTransforms for alignment_func_to_anat Node [skip ci]

* [narps_open.core] adding list_to_file function [skip ci]

* [08MQ] custom timings as input of SliceTimer [skip ci]

* Typo preprocessing

* Computing masks : output of preprocessing [skip ci]

* Computing masks : output of preprocessing [skip ci]

* Back to ApplyXFM, with no_resample option [skip ci]

* Back to ApplyXFM, with no_resample option [skip ci]

* Back to ApplyXFM, with no_resample option [skip ci]

* Mask intersection of func as input of subject_level estimate model [skip ci]

* Mask intersection of func as input of subject_level estimate model [skip ci]

* Mask intersection in group_level analysis [skip ci]

* Mask input + mask intersection inside group level analysis [skip ci]

* Adding mask to randomise Node [skip ci]

* Actually passing mask tu compute median node [skip ci]

* Actually passing mask tu compute median node [skip ci]

* Actually passing mask tu compute median node [skip ci]

* Adding mask to compute median node [skip ci]

* Adding wm sgmentation to coregistration_sbref node working with bbr cost function [skip ci]

* Adding brain extraction of sbrefs [skip ci]

* Using unthresholded wm for coregistration_sbref node [skip ci]

* Bug with WM selection [skip ci]

* Adding back remove functions for testing on 20 subjects [skip ci]

* Removing more intermediate results [skip ci]

* Force 2.0mm resampling for alignment_func_to_anat node [skip ci]

* Force 2.0mm resampling for alignment_func_to_anat node [skip ci]

* [TEST][PEP8] [skip ci]

narps_open/core/common.py

@@ -63,3 +63,24 @@ def list_intersection(list_1: list, list_2: list) -> list:
     - list, the intersection of list_1 and list_2
     """
     return [e for e in list_1 if e in list_2]
+
+def list_to_file(input_list: list, file_name: str = 'elements.tsv') -> str:
+    """
+    Create a tsv file containing elements of the input list.
+    This function is meant to be used in a Nipype Function Node.
+
+    Parameters :
+    - input_list: list
+
+    Returns:
+    - output_file: path to the created file
+    """
+    from os.path import abspath
+    output_file = abspath(file_name)
+
+    # Write un element per line
+    with open(output_file, 'w') as writer:
+        for element in input_list:
+            writer.write(f'{element}\n')
+
+    return output_file

narps_open/pipelines/__init__.py

@@ -8,7 +8,7 @@
 
 # List all the available pipelines and the corresponding class for each
 implemented_pipelines = {
-    '08MQ': None,
+    '08MQ': 'PipelineTeam08MQ',
     '0C7Q': None,
     '0ED6': None,
     '0H5E': None,

narps_open/utils/configuration/testing_config.toml

@@ -3,9 +3,9 @@ title = "Testing configuration for the NARPS open pipelines project"
 config_type = "testing"
 
 [directories]
-dataset = "run/data/ds001734/"
+dataset = "data/original/ds001734/"
 reproduced_results = "run/data/reproduced/"
-narps_results = "run/data/results/"
+narps_results = "data/results/"
 test_data = "tests/test_data/"
 test_runs = "run/"
 

tests/core/test_common.py

@@ -317,3 +317,66 @@ def test_connect_list_intersection(remove_test_dir):
         test_file_2 = join(TEMPORARY_DIR, 'test_workflow', 'node_2', '_report', 'report.rst')
         with open(test_file_2, 'r', encoding = 'utf-8') as file:
             assert f'* out_value : {output_list_2}' in file.read()
+
+    @staticmethod
+    @mark.unit_test
+    def test_node_list_to_file_1():
+        """ Test the list_to_file function as a nipype.Node """
+
+        # Inputs
+        input_list = ['001', 23.560, 'azerty', False, None]
+
+        # Create a Nipype Node using list_to_file
+        test_node = Node(Function(
+            function = co.list_to_file,
+            input_names = ['input_list'],
+            output_names = ['out_file']
+            ), name = 'test_node')
+        test_node.inputs.input_list = input_list
+        test_node.run()
+
+        # Expected output (in the Node's working directory)
+        out_file = join(test_node.output_dir(), 'elements.tsv')
+        out_list = [str(a) for a in input_list]
+
+        # Check file was created
+        assert exists(out_file)
+
+        # Check file was created
+        with open(out_file, 'r', encoding = 'utf-8') as file:
+            for list_element, file_element in zip(out_list, file.read().split('\n')):
+                assert list_element == file_element
+
+    @staticmethod
+    @mark.unit_test
+    def test_node_list_to_file_2():
+        """ Test the list_to_file function as a nipype.Node
+            Test changing name of output file
+        """
+
+        # Inputs
+        input_list = ['001', 23.560, [2.0, 1, 53, True], False, None]
+        file_name = 'custom_filename.txt'
+
+        # Create a Nipype Node using list_to_file
+        test_node = Node(Function(
+            function = co.list_to_file,
+            input_names = ['input_list', 'file_name'],
+            output_names = ['out_file']
+            ), name = 'test_node')
+        test_node.inputs.input_list = input_list
+        test_node.inputs.file_name = file_name
+        test_node.run()
+
+        # Expected output
+        out_file = join(test_node.output_dir(), file_name)
+        out_list = [str(a) for a in input_list]
+
+        # Check file was created
+        assert exists(out_file)
+
+        # Check file was created
+        with open(out_file, 'r', encoding = 'utf-8') as file:
+            for list_element, file_element in zip(out_list, file.read().split('\n')):
+                assert list_element == file_element
+

tests/pipelines/test_team_08MQ.py

@@ -0,0 +1,143 @@
+#!/usr/bin/python
+# coding: utf-8
+
+""" Tests of the 'narps_open.pipelines.team_08MQ' module.
+
+Launch this test with PyTest
+
+Usage:
+======
+    pytest -q test_team_08MQ.py
+    pytest -q test_team_08MQ.py -k <selected_test>
+"""
+from os.path import join
+
+from pytest import helpers, mark
+from numpy import isclose
+from nipype import Workflow
+from nipype.interfaces.base import Bunch
+
+from narps_open.utils.configuration import Configuration
+from narps_open.pipelines.team_08MQ import PipelineTeam08MQ
+
+class TestPipelinesTeam08MQ:
+    """ A class that contains all the unit tests for the PipelineTeam08MQ class."""
+
+    @staticmethod
+    @mark.unit_test
+    def test_create():
+        """ Test the creation of a PipelineTeam08MQ object """
+
+        pipeline = PipelineTeam08MQ()
+
+        # 1 - check the parameters
+        assert pipeline.fwhm == 6.0
+        assert pipeline.team_id == '08MQ'
+        assert pipeline.contrast_list == ['1', '2', '3']
+        assert pipeline.run_level_contasts == [
+            ('positive_effect_gain', 'T', ['gain', 'loss'], [1, 0]),
+            ('positive_effect_loss', 'T', ['gain', 'loss'], [0, 1]),
+            ('negative_effect_loss', 'T', ['gain', 'loss'], [0, -1])
+        ]
+
+        # 2 - check workflows
+        assert isinstance(pipeline.get_preprocessing(), Workflow)
+        assert isinstance(pipeline.get_run_level_analysis(), Workflow)
+        assert isinstance(pipeline.get_subject_level_analysis(), Workflow)
+
+        group_level = pipeline.get_group_level_analysis()
+        assert len(group_level) == 3
+        for sub_workflow in group_level:
+            assert isinstance(sub_workflow, Workflow)
+
+    @staticmethod
+    @mark.unit_test
+    def test_outputs():
+        """ Test the expected outputs of a PipelineTeam08MQ object """
+        pipeline = PipelineTeam08MQ()
+        # 1 - 1 subject outputs
+        pipeline.subject_list = ['001']
+        assert len(pipeline.get_preprocessing_outputs()) == 4*4
+        assert len(pipeline.get_run_level_outputs()) == 8+4*3*4
+        assert len(pipeline.get_subject_level_outputs()) == 4*3
+        assert len(pipeline.get_group_level_outputs()) == 0
+        assert len(pipeline.get_hypotheses_outputs()) == 18
+
+        # 2 - 4 subjects outputs
+        pipeline.subject_list = ['001', '002', '003', '004']
+        assert len(pipeline.get_preprocessing_outputs()) == 4*4*4
+        assert len(pipeline.get_run_level_outputs()) == (8+4*3*4)*4
+        assert len(pipeline.get_subject_level_outputs()) == 4*3*4
+        assert len(pipeline.get_group_level_outputs()) == 0
+        assert len(pipeline.get_hypotheses_outputs()) == 18
+
+    @staticmethod
+    @mark.unit_test
+    def test_subject_information():
+        """ Test the get_subject_information method """
+
+        information = PipelineTeam08MQ.get_subject_information(join(
+            Configuration()['directories']['test_data'],
+            'pipelines',
+            'events.tsv'
+            ))[0]
+
+        assert isinstance(information, Bunch)
+        assert information.conditions == ['event', 'gain', 'loss', 'response']
+
+        reference_amplitudes = [
+            [1.0, 1.0, 1.0, 1.0, 1.0],
+            [14.0, 34.0, 38.0, 10.0, 16.0],
+            [6.0, 14.0, 19.0, 15.0, 17.0],
+            [1.0, 1.0, 0.0, -1.0, -1.0]
+            ]
+        for reference_array, test_array in zip(reference_amplitudes, information.amplitudes):
+            assert isclose(reference_array, test_array).all()
+
+        reference_durations = [
+            [4.0, 4.0, 4.0, 4.0, 4.0],
+            [4.0, 4.0, 4.0, 4.0, 4.0],
+            [4.0, 4.0, 4.0, 4.0, 4.0],
+            [4.0, 4.0, 4.0, 4.0, 4.0]
+            ]
+        for reference_array, test_array in zip(reference_durations, information.durations):
+            assert isclose(reference_array, test_array).all()
+
+        reference_onsets = [
+            [4.071, 11.834, 19.535, 27.535, 36.435],
+            [4.071, 11.834, 19.535, 27.535, 36.435],
+            [4.071, 11.834, 19.535, 27.535, 36.435],
+            [4.071, 11.834, 19.535, 27.535, 36.435]
+        ]
+        for reference_array, test_array in zip(reference_onsets, information.onsets):
+            assert isclose(reference_array, test_array).all()
+
+    @staticmethod
+    @mark.unit_test
+    def test_one_sample_t_test_regressors():
+        """ Test the get_one_sample_t_test_regressors method """
+
+        regressors = PipelineTeam08MQ.get_one_sample_t_test_regressors(['001', '002'])
+        assert regressors == {'group_mean': [1, 1]}
+
+    @staticmethod
+    @mark.unit_test
+    def test_two_sample_t_test_regressors():
+        """ Test the get_two_sample_t_test_regressors method """
+
+        regressors, groups = PipelineTeam08MQ.get_two_sample_t_test_regressors(
+            ['001', '003'], # equalRange group
+            ['002', '004'], # equalIndifference group
+            ['001', '002', '003', '004'] # all subjects
+            )
+        assert regressors == dict(
+                equalRange = [1, 0, 1, 0],
+                equalIndifference = [0, 1, 0, 1]
+            )
+        assert groups == [1, 2, 1, 2]
+
+    @staticmethod
+    @mark.pipeline_test
+    def test_execution():
+        """ Test the execution of a PipelineTeam08MQ and compare results """
+        helpers.test_pipeline_evaluation('08MQ')

tests/test_data/pipelines/events.tsv

@@ -0,0 +1,6 @@
+onset	duration	gain	loss	RT	participant_response
+4.071	4	14	6	2.388	weakly_accept
+11.834	4	34	14	2.289	strongly_accept
+19.535	4	38	19	0	NoResp
+27.535	4	10	15	2.08	strongly_reject
+36.435	4	16	17	2.288	weakly_reject

tests/test_runner.py

@@ -195,7 +195,7 @@ def test_create():
 
         # 3 - Instantiate a runner with a not implemented team id
         with raises(NotImplementedError):
-            PipelineRunner('08MQ')
+            PipelineRunner('1K0E')
 
         # 4 - Instantiate a runner with an implemented team id
         runner = PipelineRunner('2T6S')
@@ -204,7 +204,7 @@ def test_create():
 
         # 5 - Modify team id for an existing runner (with a not implemented team id)
         with raises(NotImplementedError):
-            runner.team_id = '08MQ'
+            runner.team_id = '1K0E'
 
     @staticmethod
     @mark.unit_test