research-article
Authors: Jun Lyu, Xiudong Chen, Salman A. AlQahtani, and M. Shamim Hossain
Volume 107, Issue C
Published: 02 July 2024 Publication History
- 0citation
- 0
- Downloads
Metrics
Total Citations0Total Downloads0Last 12 Months0
Last 6 weeks0
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
- View Options
- References
- Media
- Tables
- Share
Abstract
Magnetic Resonance Imaging (MRI) is a pivotal neuroimaging technique capable of generating images with various contrasts, known as multi-modal images. The integration of these diverse modalities is essential for improving model performance across various tasks. However, in real clinical scenarios, acquiring MR images for all modalities is frequently hindered by factors such as patient comfort and scanning costs. Therefore, effectively fusing different modalities to synthesize missing modalities has become a research hot-spot in the field of smart healthcare, particularly in the context of the Internet of Medical Things (IoMT). In this study, we introduce a multi-modal coordinated fusion network (MCF-Net) with Patch Complementarity Pre-training. This network leverages the complementarity and correlation between different modalities to make the fusion of multi-modal MR images, addressing challenges in the IoMT. Specifically, we first employ a Patch Complementarity Mask Autoencoder (PC-MAE) for self-supervised pre-training. The complementarity learning mechanism is introduced to align masks and visual annotations between two modalities. Subsequently, a dual-branch MAE architecture and a shared encoder–decoder are adopted to facilitate cross-modal interactions within mask tokens. Furthermore, during the fine-tuning phase, we incorporate an Attention-Driven Fusion (ADF) module into the MCF-Net. This module synthesizes missing modal images by fusion of multi-modal features from the pre-trained PC-MAE encoder. Additionally, we leverage the pre-trained encoder to extract high-level features from both synthetic and corresponding real images, ensuring consistency throughout the training process. Our experimental findings showcase a notable enhancement in performance across various modalities with our fusion method, outperforming state-of-the-art techniques.
Highlights
•
Propose a multi-modal medical image fusion framework with patch complementary pre-training
•
Design a novel masking alignment strategy to learn complementary information between modalities.
•
Introduce an attention-driven fusion module to aggregate multi-modal features.
References
[1]
Zhou Tao, Liu Mingxia, Fu Huazhu, Wang Jun, Shen Jianbing, Shao Ling, Shen Dinggang, Deep multi-modal latent representation learning for automated dementia diagnosis, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2019, pp. 629–638.
[2]
Muhammad Ghulam, Alshehri Fatima, Karray Fakhri, El Saddik Abdulmotaleb, Alsulaiman Mansour, Falk Tiago H, A comprehensive survey on multimodal medical signals fusion for smart healthcare systems, Inf. Fusion 76 (2021) 355–375.
Digital Library
[3]
Zhang Pengfei, Li Tianrui, Wang Guoqiang, Wang Dexian, Lai Pei, Zhang Fan, A multi-source information fusion model for outlier detection, Inf. Fusion 93 (2023) 192–208.
[4]
Zhang Hao, Xu Han, Tian Xin, Jiang Junjun, Ma Jiayi, Image fusion meets deep learning: A survey and perspective, Inf. Fusion 76 (2021) 323–336.
Digital Library
[5]
Sevetlidis Vasileios, Giuffrida Mario Valerio, Tsaftaris Sotirios A, Whole image synthesis using a deep encoder-decoder network, in: Simulation and Synthesis in Medical Imaging: First International Workshop, SASHIMI 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings 1, Springer, 2016, pp. 127–137.
[6]
Bavirisetti Durga Prasad, Dhuli Ravindra, Two-scale image fusion of visible and infrared images using saliency detection, Infrared Phys. Technol. 76 (2016) 52–64.
[7]
Dar Salman UH, Yurt Mahmut, Karacan Levent, Erdem Aykut, Erdem Erkut, Cukur Tolga, Image synthesis in multi-contrast MRI with conditional generative adversarial networks, IEEE Trans. Med. Imaging 38 (10) (2019) 2375–2388.
[8]
Fu Yu, Wu Xiao-Jun, Durrani Tariq, Image fusion based on generative adversarial network consistent with perception, Inf. Fusion 72 (2021) 110–125.
[9]
Wen Wei, Emilie Poirion, Benedetta Bodini, Stanley Durrleman, Olivier Colliot, Bruno Stankoff, Nicholas Ayache, FLAIR MR image synthesis by using 3D fully convolutional networks for multiple sclerosis, in: ISMRM-ESMRMB 2018-Joint Annual Meeting, 2018, pp. 1–6.
[10]
Wang Guanhua, Gong Enhao, Banerjee Suchandrima, Martin Dann, Tong Elizabeth, Choi Jay, Chen Huijun, Wintermark Max, Pauly John M, Zaharchuk Greg, Synthesize high-quality multi-contrast magnetic resonance imaging from multi-echo acquisition using multi-task deep generative model, IEEE Trans. Med. Imaging 39 (10) (2020) 3089–3099.
[11]
Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2223–2232.
[12]
Yang Xin, Lin Yi, Wang Zhiwei, Li Xin, Cheng Kwang-Ting, Bi-modality medical image synthesis using semi-supervised sequential generative adversarial networks, IEEE J. Biomed. Health Inform. 24 (3) (2019) 855–865.
[13]
Olut Sahin, Sahin Yusuf H., Demir Ugur, Unal Gozde, Generative adversarial training for MRA image synthesis using multi-contrast MRI, in: PRedictive Intelligence in MEdicine: First International Workshop, PRIME 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 1, Springer, 2018, pp. 147–154.
[14]
Zhou Tao, Fu Huazhu, Chen Geng, Shen Jianbing, Shao Ling, Hi-net: Hybrid-fusion network for multi-modal MR image synthesis, IEEE Trans. Med. Imaging 39 (9) (2020) 2772–2781.
Digital Library
[15]
Hossain M.S., et al., Smart healthcare monitoring: A voice pathology detection paradigm for smart cities, Multimedia Syst. 25 (5) (2019) 565–575.
[16]
Umirzakova Sabina, Ahmad Shabir, Khan Latif U, Whangbo Taegkeun, Medical image super-resolution for smart healthcare applications: A comprehensive survey, Inf. Fusion (2023).
[17]
Tao Xiaohui, Velasquez Juan D., Multi-source information fusion for smart health with artificial intelligence, Inf. Fusion 83 (2022) 93–95.
[18]
Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, Bengio Yoshua, Generative adversarial nets, Advances in Neural Information Processing Systems, vol. 27, 2014.
[19]
Dar Salman UH, Yurt Mahmut, Karacan Levent, Erdem Aykut, Erdem Erkut, Cukur Tolga, Image synthesis in multi-contrast MRI with conditional generative adversarial networks, IEEE Trans. Med. Imaging 38 (10) (2019) 2375–2388.
[20]
Wolterink Jelmer M, Dinkla Anna M, Savenije Mark HF, Seevinck Peter R, van den Berg Cornelis AT, Išgum Ivana, Deep MR to CT synthesis using unpaired data, in: Simulation and Synthesis in Medical Imaging: Second International Workshop, SASHIMI 2017, Held in Conjunction with MICCAI 2017, QuÉBec City, QC, Canada, September 10, 2017, Proceedings 2, Springer, 2017, pp. 14–23.
[21]
Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2223–2232.
[22]
Wen Wei, Emilie Poirion, Benedetta Bodini, Stanley Durrleman, Olivier Colliot, Bruno Stankoff, Nicholas Ayache, FLAIR MR image synthesis by using 3D fully convolutional networks for multiple sclerosis, in: ISMRM-ESMRMB 2018-Joint Annual Meeting, 2018, pp. 1–6.
[23]
Huang Yawen, Shao Ling, Frangi Alejandro F., DOTE: Dual convolutional filter learning for super-resolution and cross-modality synthesis in MRI, in: Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20, Springer, 2017, pp. 89–98.
[24]
Shen Liyue, Zhu Wentao, Wang Xiaosong, Xing Lei, Pauly John M, Turkbey Baris, Harmon Stephanie Anne, Sanford Thomas Hogue, Mehralivand Sherif, Choyke Peter L, et al., Multi-domain image completion for random missing input data, IEEE Trans. Med. Imaging 40 (4) (2020) 1113–1122.
[25]
Wang Yan, Zhou Luping, Yu Biting, Wang Lei, Zu Chen, Lalush David S, Lin Weili, Wu Xi, Zhou Jiliu, Shen Dinggang, 3D auto-context-based locality adaptive multi-modality GANs for PET synthesis, IEEE Trans. Med. Imaging 38 (6) (2018) 1328–1339.
[26]
Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N, Kaiser Łukasz, Polosukhin Illia, Attention is all you need, Advances in Neural Information Processing Systems, vol. 30, 2017.
[27]
Chen Jieneng, Lu Yongyi, Yu Qihang, Luo Xiangde, Adeli Ehsan, Wang Yan, Lu Le, Yuille Alan L, Zhou Yuyin, Transunet: Transformers make strong encoders for medical image segmentation, 2021, arXiv preprint arXiv:2102.04306.
[28]
Chen Junyu, He Yufan, Frey Eric C., Li Ye, Du Yong, Vit-v-net: Vision transformer for unsupervised volumetric medical image registration, 2021, arXiv preprint arXiv:2104.06468.
[29]
Dalmaz Onat, Yurt Mahmut, Çukur Tolga, ResViT: Residual vision transformers for multimodal medical image synthesis, IEEE Trans. Med. Imaging 41 (10) (2022) 2598–2614.
[30]
Peng Bo, Liu Bingzheng, Bin Yi, Shen Lili, Lei Jianjun, Multi-modality mr image synthesis via confidence-guided aggregation and cross-modality refinement, IEEE J. Biomed. Health Inf. 26 (1) (2021) 27–35.
[31]
Yan Shouang, Wang Chengyan, Chen Weibo, Lyu Jun, Swin transformer-based GAN for multi-modal medical image translation, Front. Oncol. 12 (2022).
[32]
Liu Jiang, Pasumarthi Srivathsa, Duffy Ben, Gong Enhao, Datta Keshav, Zaharchuk Greg, One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation, IEEE Trans. Med. Imaging (2023).
[33]
Li Yonghao, Zhou Tao, He Kelei, Zhou Yi, Shen Dinggang, Multi-scale transformer network with edge-aware pre-training for cross-modality MR image synthesis, IEEE Trans. Med. Imaging (2023).
[34]
Li Yan, Xu Sisi, Chen Haibin, Sun Ying, Bian Jing, Guo Shuanshuan, Lu Yao, Qi Zhenyu, CT synthesis from multi-sequence MRI using adaptive fusion network, Comput. Biol. Med. 157 (2023).
[35]
Li Yan, Xu Sisi, Lu Yao, Qi Zhenyu, CT synthesis from MRI with an improved multi-scale learning network, Front. Phys. 11 (2023).
[36]
Hardoon David R., Szedmak Sandor, Shawe-Taylor John, Canonical correlation analysis: An overview with application to learning methods, Neural Comput. 16 (12) (2004) 2639–2664.
Digital Library
[37]
Lin Yen-Yu, Liu Tyng-Luh, Fuh Chiou-Shann, Multiple kernel learning for dimensionality reduction, IEEE Trans. Pattern Anal. Mach. Intell. 33 (6) (2010) 1147–1160.
Digital Library
[38]
Zhou Tao, Thung Kim-Han, Zhu Xiaofeng, Shen Dinggang, Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis, Hum. Brain Mapp. 40 (3) (2019) 1001–1016.
[39]
Hou Ming, Tang Jiajia, Zhang Jianhai, Kong Wanzeng, Zhao Qibin, Deep multimodal multilinear fusion with high-order polynomial pooling, Adv. Neural Inf. Process. Syst. 32 (2019).
[40]
Yu Biting, Zhou Luping, Wang Lei, Fripp Jurgen, Bourgeat Pierrick, 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation, in: 2018 IEEE 15th International Symposium on Biomedical Imaging, ISBI 2018, IEEE, 2018, pp. 626–630.
[41]
Wei Wen, Poirion Emilie, Bodini Benedetta, Durrleman Stanley, Colliot Olivier, Stankoff Bruno, Ayache Nicholas, Fluid-attenuated inversion recovery MRI synthesis from multisequence MRI using three-dimensional fully convolutional networks for multiple sclerosis, J. Med. Imaging 6 (1) (2019) 014005.
[42]
Menze Bjoern H, Jakab Andras, Bauer Stefan, Kalpathy-Cramer Jayashree, Farahani Keyvan, Kirby Justin, Burren Yuliya, Porz Nicole, Slotboom Johannes, Wiest Roland, et al., The multimodal brain tumor image segmentation benchmark (BRATS), IEEE Trans. Med. Imaging 34 (10) (2014) 1993–2024.
[43]
Zhang Xuzhe, He Xinzi, Guo Jia, Ettehadi Nabil, Aw Natalie, Semanek David, Posner Jonathan, Laine Andrew, Wang Yun, PTNet: A high-resolution infant MRI synthesizer based on transformer, 2021, arXiv preprint arXiv:2105.13993.
[44]
Yu Biting, Zhou Luping, Wang Lei, Shi Yinghuan, Fripp Jurgen, Bourgeat Pierrick, Ea-GANs: Edge-aware generative adversarial networks for cross-modality MR image synthesis, IEEE Trans. Med. Imaging 38 (7) (2019) 1750–1762.
Recommendations
- A comprehensive survey on multimodal medical signals fusion for smart healthcare systems
Highlights
- A comprehensive survey on IoMT-based fusion for healthcare is presented.
- ...
Abstract
Smart healthcare is a framework that utilizes technologies such as wearable devices, the Internet of Medical Things (IoMT), sophisticated machine learning algorithms, and wireless communication technology to seamlessly access health ...
Read More
- Multi-modal medical image fusion in NSST domain for internet of medical things
Abstract
The Internet of Medical Things (IoMT) has included a new layer for development and smart infrastructure growth in the medical field. Besides, the medical data on IoMT systems are constantly expanding due to the rising peripherals in the health ...
Read More
- A survey of multimodal information fusion for smart healthcare: Mapping the journey from data to wisdom
Abstract
Multimodal medical data fusion has emerged as a transformative approach in smart healthcare, enabling a comprehensive understanding of patient health and personalized treatment plans. In this paper, a journey from data to information to knowledge ...
Read More
Comments
Information & Contributors
Information
Published In
Information Fusion Volume 107, Issue C
Jul 2024
522 pages
ISSN:1566-2535
Issue’s Table of Contents
Elsevier B.V.
Publisher
Elsevier Science Publishers B. V.
Netherlands
Publication History
Published: 02 July 2024
Author Tags
- Multi-modal medical image fusion
- Smart healthcare
- Internet of medical things (IoMT)
- Magnetic resonance imaging (MRI)
Qualifiers
- Research-article
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
Total Citations
Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Other Metrics
View Author Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Publication
Media
Figures
Other
Tables
