MMIF: Multimodal Medical Image Fusion Network Based on Multi-Scale Hybrid Attention

Abstract

Multimodal image fusion plays an important role in image analysis and applications. Multimodal medical image fusion helps to combine contrast features from two or more input imaging modalities to represent fused information in a single image. One of the critical clinical applications of medical image fusion is to fuse anatomical and functional modalities for rapid diagnosis of malignant tissues. This paper proposes a multimodal medical image fusion network (MMIF-Net) based on multiscale hybrid attention. The method first decomposes the original image to obtain the low-rank and significant parts. Then, to utilize the features at different scales, we add a multiscale mechanism that uses three filters of different sizes to extract the features in the encoded network. Also, a hybrid attention module is introduced to obtain more image details. Finally, the fused images are reconstructed by decoding the network. We conducted experiments with clinical images from brain computed tomography/magnetic resonance. The experimental results show that the multimodal medical image fusion network method based on multiscale hybrid attention works better than other advanced fusion methods.