Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH]: DICOM to NIFTI BIDS conversion with Heudiconv #9

Merged
merged 92 commits into from
Jul 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
aaa2dd0
[ENH]: DICOM to NIFTI BIDS conversion with heuristic generated from c…
alexpron Mar 22, 2024
35a8d71
- added temporary directories to download in DICOM archives and extra…
alexpron Mar 25, 2024
b1f0d5a
Merge branch 'Inria-Empenn:main' into heudiconv
alexpron Mar 25, 2024
a66b273
- renamed few modality suffixes so that it fits BIDS spec
alexpron Mar 25, 2024
4c70ea9
Merge remote-tracking branch 'origin/heudiconv' into heudiconv
alexpron Mar 25, 2024
0a12916
change main script name
alexpron Mar 26, 2024
7f78df1
+ modified default heudiconv StudyID grouping (used all) to handle mu…
alexpron Mar 27, 2024
2c40721
Merge branch 'bids-1.9.0-compliance' into heudiconv
alexpron Mar 28, 2024
4aa85ce
+
alexpron Mar 28, 2024
4656cc5
clean parser options
alexpron Apr 8, 2024
d07faee
clean parser options
alexpron Apr 9, 2024
1ed0c69
added dcm2niix executable check
alexpron Apr 15, 2024
0139aed
added back automri support option
alexpron Apr 22, 2024
59a2f09
Revert "Merge branch 'bids-1.9.0-compliance' into heudiconv"
alexpron Apr 22, 2024
0fca77a
added main dcm2niix configuration options into nipype format + docume…
alexpron Apr 22, 2024
a5b9d31
fix authors
alexpron Apr 22, 2024
295bd52
[FIX]: typo
alexpron May 21, 2024
5064edb
[ENH]: not working hack for heudiconv support
alexpron May 21, 2024
6ff27a5
[LINT]: black linting these files
alexpron May 21, 2024
ad7a639
[FIX] modified heuristic file to support automri
alexpron May 22, 2024
7b87841
[FIX] fixed typo introduced by black linting in json file
alexpron May 22, 2024
8966d63
[FIX] fixed typo introduced by black linting in json file
alexpron May 22, 2024
21cbdc2
[ENH]: v2 of automri generation. Not to be used (back up solution) us…
alexpron May 22, 2024
3c46df2
[FIX]: removed automri support and linting
alexpron May 28, 2024
c3c03cf
[FIX]: removed temporary directories (not supported by python <3.11) …
alexpron Jun 3, 2024
005c9b7
[FIX]:
alexpron Jun 3, 2024
b3dfb28
[ENH]: DICOM to NIFTI BIDS conversion with heuristic generated from c…
alexpron Mar 22, 2024
b7384c6
- added temporary directories to download in DICOM archives and extra…
alexpron Mar 25, 2024
b7a8b3f
- renamed few modality suffixes so that it fits BIDS spec
alexpron Mar 25, 2024
52aa9cb
change main script name
alexpron Mar 26, 2024
ce3722a
+ modified default heudiconv StudyID grouping (used all) to handle mu…
alexpron Mar 27, 2024
12e1e43
proposal on the s2b_example_config.json to enhance BIDS compatibility
alexpron Mar 25, 2024
e088703
fix underscores in filename suffixes
alexpron Mar 26, 2024
1992915
fix underscores in filename suffixes
alexpron Mar 26, 2024
4afe579
Revert "fix underscores in filename suffixes"
alexpron Mar 26, 2024
a7ac1eb
fix underscores in filename suffixes
alexpron Mar 26, 2024
29326f6
+
alexpron Mar 28, 2024
58e17d2
clean parser options
alexpron Apr 8, 2024
86fced9
clean parser options
alexpron Apr 9, 2024
f1a1878
added dcm2niix executable check
alexpron Apr 15, 2024
0cf37dd
added back automri support option
alexpron Apr 22, 2024
ed2b86b
Revert "Merge branch 'bids-1.9.0-compliance' into heudiconv"
alexpron Apr 22, 2024
1bd43df
added main dcm2niix configuration options into nipype format + docume…
alexpron Apr 22, 2024
886eb1d
fix authors
alexpron Apr 22, 2024
3202715
[FIX]: typo
alexpron May 21, 2024
4b4dc06
[ENH]: not working hack for heudiconv support
alexpron May 21, 2024
b13bd8e
[LINT]: black linting these files
alexpron May 21, 2024
0c7c351
[FIX] modified heuristic file to support automri
alexpron May 22, 2024
77f8e9e
[FIX] fixed typo introduced by black linting in json file
alexpron May 22, 2024
9feb924
[FIX] fixed typo introduced by black linting in json file
alexpron May 22, 2024
2672fc4
[ENH]: v2 of automri generation. Not to be used (back up solution) us…
alexpron May 22, 2024
03618e3
[FIX]: removed automri support and linting
alexpron May 28, 2024
0436e2d
[FIX]: removed temporary directories (not supported by python <3.11) …
alexpron Jun 3, 2024
36e311f
[FIX]:
alexpron Jun 3, 2024
8c3e70a
Merge remote-tracking branch 'origin/heudiconv' into heudiconv
alexpron Jun 3, 2024
c6a1570
Merge branch 'main' into heudiconv
alexpron Jun 3, 2024
f5d5029
[ENH]: added temporary directories
alexpron Jun 27, 2024
1534a3e
[ENH]: check BIDS mapping beforehand
alexpron Jun 27, 2024
ce7019c
[ENH]: shanoir2bids_heudiconv is now shanoir2bids
alexpron Jul 15, 2024
fced7df
[ENH]: enhanced documentation for shanoir2bids
alexpron Jul 16, 2024
714720d
[ENH]: added installation instructions and metadata in a pyproject.to…
alexpron Jul 16, 2024
eb348a6
[ENH]: fixed dependencies and build backend issue
alexpron Jul 16, 2024
b255d2e
[ENH]: fixed dependencies and build backend issue
alexpron Jul 16, 2024
b39ef9a
[ENH]: added authors emails
alexpron Jul 16, 2024
85f1efa
[ENH]: adapted installations instructions in the README to new mechanism
alexpron Jul 16, 2024
894a694
[ENH]: added CI to test shanoir downloader basic installation
alexpron Jul 16, 2024
53b3dd6
[ENH]: restrained to ubuntu in the first time
alexpron Jul 16, 2024
b9a5969
[BF]: try fix bug with python 3.11 installation
alexpron Jul 16, 2024
758dcf0
[BF]: fix syntax
alexpron Jul 16, 2024
7f04e6b
[BF]: restrained python version to python<3.11
alexpron Jul 16, 2024
2c99028
[BF]: added environment.yml corresponding to requirements.txt
alexpron Jul 18, 2024
9de8f02
[BF]: fix pip and conda installation syntax
alexpron Jul 18, 2024
2c18191
[BF]: fix pip and conda installation syntax
alexpron Jul 18, 2024
6e4b1d3
[BF]: fix pip and conda installation syntax
alexpron Jul 18, 2024
e123685
[BF]: fix pip and conda installation syntax
alexpron Jul 18, 2024
3017c11
[BF]: fix conda syntax for build
alexpron Jul 18, 2024
36d9489
[BF]: updated installation instructions
alexpron Jul 18, 2024
303f695
[BF]: updated conda installation and instructions so that pip package…
alexpron Jul 19, 2024
20faca1
+
alexpron Mar 28, 2024
a67dd06
clean parser options
alexpron Apr 9, 2024
e2485f3
added main dcm2niix configuration options into nipype format + docume…
alexpron Apr 22, 2024
51a662a
[LINT]: black linting these files
alexpron May 21, 2024
3a84b3d
[FIX] fixed typo introduced by black linting in json file
alexpron May 22, 2024
898ee62
[ENH]: DICOM to NIFTI BIDS conversion with heuristic generated from c…
alexpron Mar 22, 2024
ff5aee9
change main script name
alexpron Mar 26, 2024
669b85e
[FIX]: typo
alexpron May 21, 2024
b27d404
[ENH]: check BIDS mapping beforehand
alexpron Jun 27, 2024
5b16489
[ENH]: enhanced documentation for shanoir2bids
alexpron Jul 16, 2024
3765d05
[BF]: fixed typos config file after rebase
alexpron Jul 22, 2024
b066660
[ENH] added explicit conda command installation
alexpron Jul 22, 2024
ac64c3e
[BF]: added bids-validator package
alexpron Jul 22, 2024
d98a8bb
Merge branch 'main' into heudiconv
alexpron Jul 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 32 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,20 @@ conda install -c conda-forge heudiconv git-annex=*=alldep* datalad
```
## Usage

There are three scripts to download datasets:
- `shanoir_downloader.py` simply downloads datasets from a id, a list of ids, or directly from a [shanoir search as on the shanoir solr search page](https://shanoir.irisa.fr/shanoir-ng/solr-search),
- `shanoir_downloader_check.py` is a more complete tool ; it enables to download datasets (from a csv or excel file containing a list of dataset ids, or directly from a [shanoir search as on the shanoir solr search page](https://shanoir.irisa.fr/shanoir-ng/solr-search)), verify their content, anonymize them and / or encrypt them.
- `shanoir2bids.py` uses `shanoir_downloader.py` to download Shanoir datasets and reorganises them into a BIDS data structure that is specified by the user with a `.json` configuration file. An example of configuration file is provided `s2b_example_config.json`.
## How to download Shanoir datasets ?

`shanoir_downloader_check.py` creates two files in the output folder:
- `downloaded_datasets.csv` records the successfully downloaded datasets,
- `missing_datasets.csv` records the datasets which could not be downloaded.
There are three scripts to download datasets from a Shanoir instance:

1.`shanoir_downloader.py`: downloads datasets from a id, a list of ids, or directly from a [shanoir search as on the shanoir solr search page](https://shanoir.irisa.fr/shanoir-ng/solr-search),

2.`shanoir_downloader_check.py`: a more complete tool ; it enables to download datasets (from a csv or excel file containing a list of dataset ids, or directly from a [shanoir search as on the shanoir solr search page](https://shanoir.irisa.fr/shanoir-ng/solr-search)), verify their content, anonymize them and / or encrypt them.

3.`shanoir2bids.py`: Download datasets from Shanoir in DICOM format and convert them into datalad datasets in BIDS format. The conversion is parameterised by a configuration file in `.json` format.

### `shanoir_downloader_check.py`
- `shanoir_downloader_check.py` creates two files in the output folder:
- `downloaded_datasets.csv` records the successfully downloaded datasets,
- `missing_datasets.csv` records the datasets which could not be downloaded.

With those two files, `shanoir_downloader_check.py` is able to resume a download session (the downloading can be interrupted any time, the tool will not redownload datasets which have already been downloaded).

Expand All @@ -53,7 +59,11 @@ See `python shanoir_downloader_check.py --help` for more information.

You might want to skip the anonymization process and the encryption process with the `--skip_anonymization` and `--skip_encryption` arguments respectively (or `-sa` and `-se`).

For `shanoir2bids.py`, a configuration file must be provided to transform a Shanoir dataset into a BIDS dataset.

### `shanoir2bids.py`

A `.json` configuration file must be provided to transform a Shanoir dataset into a BIDS dataset.

```
-----------------------------[.json configuration file information]-------------------------------
This file will tell the script what Shanoir datasets should be downloaded and how the data will be organised.
Expand All @@ -64,31 +74,40 @@ The dictionary in the json file must have four keys :
"data_to_bids": list of dict, each dictionary specifies datasets to download and BIDS format with the following keys :
-> "datasetName": str, Shanoir name for the sequence to search
-> "bidsDir" : str, BIDS subdirectory sequence name (eg : "anat", "func" or "dwi", ...)
-> "bidsName" : str, BIDS sequence name (eg: "t1w-mprage", "t2-hr", "cusp66-ap-b0", ...)

-> "bidsName" : str, BIDS sequence name (eg: "t1w", "acq-b0_dir-AP", ...)
```
Please refer to the [BIDS starter kit](https://bids-standard.github.io/bids-starter-kit/folders_and_files/files.html)
for exhaustive templates of filenames. A BIDS compatible example is provided in the file `s2b_example_config.json`.

An example is provided in the file `s2b_example_config.json`.

To download longitudinal data, a key `session` and a new entry `bidsSession` in `data_to_bids` dictionaries should be defined in the JSON configuration files. Of note, only one session can be downloaded at once. Then, the key `session` is just a string, not a list as for subjects.


### Example usage

### Download Examples
#### Raw download

To download datasets, verify the content of them, anonymize them and / or encrypt them you can use a command like:

`python shanoir_downloader_check.py -u username -d shanoir.irisa.fr -ids path/to/datasets_to_download.csv -of path/to/output/folder/ -se -lf path/to/downloads.log`

The `example_input_check.csv` file in this repository is an example input file (the format of the `datasets_to_download.csv` file should be the same).


#### Solr search download

You can also download datasets from a [SolR search](https://shanoir.irisa.fr/shanoir-ng/solr-search) as on the website:

`python shanoir_downloader.py -u amasson -d shanoir.irisa.fr -of /data/amasson/test/shanoir_test4 --search_text "FLAIR" -p 1 -s 2 `

where `--search_text` is the string you would use on [the SolR search page](https://shanoir.irisa.fr/shanoir-ng/solr-search) (for example `(subjectName:(CT* OR demo*) AND studyName:etude test) OR datasetName:*flair*`). More information on the info box of the SolR search page.

`python shanoir2bids.py -c s2b_example_config.json -d my_download_dir` will download Shanoir files identified in the configuration file and saves them as a BIDS data structure into `my_download_dir`
#### BIDS download
`python shanoir2bids.py -j s2b_example_config.json -of my_download_dir --outformat nifti` will download Shanoir datasets identified in the configuration file saves them as DICOM and convert them into a BIDS datalad dataset into `my_download_dir`.

## About Solr Search

### Search usage

The `--search_text` and `--expert_mode` arguments work as on the [Shanoir search page](https://shanoir.irisa.fr/shanoir-ng/solr-search).

Expand Down
8 changes: 7 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,13 @@ readme = "README.md"
license = {file = "LICENSE"}
keywords = ["Shanoir", "DICOM", "NIFTI", "BIDS"]
classifiers = [
"Programming Language :: Python"

"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10"

]

[project.urls]
Expand Down
36 changes: 23 additions & 13 deletions s2b_example_config.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,29 @@
{
"study_name": "Aneravimm",
"subjects": ["VS_Aneravimm_010", "VS_Aneravimm_011"],
"data_to_bids":
[
{"datasetName": "t1_mprage_sag_p2_iso", "bidsDir": "anat", "bidsName": "acq-mprage_T1w"},
"study_name": "Aneravimm",
"subjects": ["VS_Aneravimm_010", "VS_Aneravimm_011"],
"data_to_bids": [
{"datasetName": "t1_mprage_sag_p2_iso", "bidsDir": "anat", "bidsName": "acq-mprage_T1w"},
{"datasetName": "t2_tse_HR_cor_MTL", "bidsDir": "anat", "bidsName": "acq-hr_T2w"},
{"datasetName": "Resting State_bold AP 1.6mm HIPP", "bidsDir": "func", "bidsName": "task-restingState_acq-hipp_dir-AP_bold"},
{"datasetName": "Resting State_bold AP 1.6mm HIPP", "bidsDir": "func", "bidsName": "task-restingState_dir-AP_bold"},
{"datasetName": "Diff cusp66 b3000 AP 1.5mm", "bidsDir": "dwi", "bidsName": "acq-b3000_dir-AP_dwi"},
{"datasetName": "Diff cusp66 b0 PA 1.5mm", "bidsDir": "dwi", "bidsName": "acq-b0_dir-AP_dwi"},
{"datasetName": "Diff cusp66 b0 PA 1.5mm", "bidsDir": "dwi", "bidsName": "acq-b0_dir-PA_dwi"}
],
"dcm2niix":"/home/qduche/Software/dcm2niix_lnx/dcm2niix",
"dcm2niix_options": "-v 0 -z y",
"find_and_replace_subject":
[
{"find":"VS_Aneravimm_", "replace": "VS"},
{"find":"Vs_Aneravimm_", "replace": "VS"}
]
"dcm2niix": "/home/alpron/softs/miniconda3/envs/test-env/bin/dcm2niix",
"dcm2niix_options_comment": "dcm2niix configuration options in the nipype format (see https://nipype.readthedocs.io/en/latest/api/generated/nipype.interfaces.dcm2nii.html)",
"dcm2niix_options": {
"bids_format": true,
"anon_bids": true,
"compress": "y",
"compression": 5,
"crop": false,
"has_private": false,
"ignore_deriv": false,
"single_file": false,
"verbose": false
},
"find_and_replace_subject": [
{"find": "VS_Aneravimm_", "replace": "VS"},
{"find": "Vs_Aneravimm_", "replace": "VS"}
]
}
Loading