combining image with tabular data in one model (integrating demographic features with image features)

[nouranik]

Dear members, thank you for this great job!

I would need to add some demographic, and clinicopathological data to the image classification task that I'm working on. how can I add these tabular data to the training process??
in another word, how to Concatenate layer output with additional input ?

thank you for your response

[Nic-Ma]

Hi @nouranik ,

Thanks for your interest here.
I think you are working on some multi-modal task?
So you can put the extra data in a dictionary, like: {"image": "/data/imga1.png", "extra1": 123, "extra2": 456}.
Then apply MONAI dictionary transforms with specified keys.

If you want to load data from CSV files, you can take a look at the CSVDataset tutorial:
https://github.com/Project-MONAI/tutorials/blob/master/modules/csv_datasets.ipynb
And the TCIA example tutorial:
https://github.com/Project-MONAI/tutorials/blob/master/modules/tcia_csv_processing.ipynb

Thanks.

[nouranik]

thanks for your prompt reply!
is there any training tutorial using CSV data loader?
thanks in advance

[Nic-Ma]

Hi @nouranik ,

We don't have such a training example so far, but I think the data loading part is same as above tutorials, and you define your own network to handle multi-modal data.

Thanks.

[nouranik]

there is a library called niftytorch for neuroimaging that used MONAI library and added demographic data to the classification model but unfortunately, I could not find the codes!!
I would appreciate it if you could provide a simple method for concatenating the extra features to the last layer of the Regressor network!!!
many many thanks!

[Nic-Ma]

Hi @wyli @ericspod ,

Do you know anything about this niftytorch package? And maybe you have some training example for multi-modal?

Thanks in advance.

[ericspod]

I haven't seen this library before but yes it doesn't look fully released yet.

[nouranik]

I could not use it since it seems it is not ready yet!
if you could kindly provide a simple example based on the IXI brain dataset (https://brain-development.org/ixi-dataset/) that you have gender classification tutorial on that in order to do brain age regression model using mixed inputs
(input=multimodal MR images + sex)

I just want to concatenate the sex information to the last layer of my Regressor model!
that would be perfect if you could provide this modification or even simple demo for updating the forward model

thanks!

[wyli]

there is a prototype by @nvahmadi https://github.com/nvahmadi/MONAI_multimodal/blob/main/demo_monai_multimodal_crisismmd.ipynb
might be useful in this case.

[nvahmadi]

Hi @wyli and @nouranik,
thanks @wyli for tagging me. The notebook you linked is actually for image+text/NLP multimodal learning on CrisisMMD dataset. Yesterday, I also uploaded an example notebook on image+EHR data (sex = 1-element EHR "vector") on ai4covid hackacthon data. This notebook is here:
https://github.com/nvahmadi/MONAI_multimodal/blob/main/demo_monai_multimodal_ai4covid.ipynb
Please note that the ai4covid notebook still needs some polishing (ETA next week). Compared to the CrisisMMD notebook, the data is also not readily downloadable. You'd need to register with the hackathon here: https://ai4covid-hackathon.it/register/
Even without the data, the example should serve as an example of how to accomplish image+EHR learning in MONAI. Let me know if you have questions.
Thanks!