Multimodal Fusion Learning with Dual Attention for Medical Imaging

TL;DR

This work tackles the challenge of generalizable multimodal fusion in medical imaging by introducing DRIFA-Net, which combines two complementary attention mechanisms—MFA for modality-specific refinement and MIFA for cross-modal fusion—within a target-specific multitask learning framework. An uncertainty quantification phase based on ensemble Monte Carlo dropout provides calibrated risk estimates for predictions. Empirical results across five public datasets spanning dermoscopy, pap smear, MRI, and CT demonstrate consistent outperformance of state-of-the-art multimodal fusion methods, with ablations confirming the value of both MFA and MIFA. The proposed approach offers a robust, interpretable, and generalizable solution for integrating heterogeneous health-record modalities in medical-image classification tasks, with practical implications for risk-aware clinical decision support.

Abstract

Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However, existing methods face three key limitations. First, they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second, they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally, they typically rely on a single attention mechanism, missing the benefits of multiple attention strategies within and across various modalities. To address these issues, this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules, i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network, forming a multimodal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality, such as dermoscopy, pap smear, MRI, and CT-scan, whereas multimodal information fusion attention module learns more refined multimodal shared representations, improving the network's generalization across multiple tasks and enhancing overall performance. Additionally, to estimate the uncertainty of DRIFA-Net predictions, we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.

Figures (5)

• Figure 1: Detailed architecture of DRIFA-Net. Key components include: (A) the target-specific multimodal fusion learning (TMFL) phase, followed by (B) an uncertainty quantification (UQ) phase. TMFL phase comprises a robust residual attention (RRA) block, shown in (C), and utilizes multi-branch fusion attention (MFA), an additional MFA module for further refinement of local representations, a multimodal information fusion attention (MIFA) module for improved multimodal representation learning, and multitask learning (MTL) for handling multiple classification tasks. During (UQ) phase, the reliability of DRIFA-Net predictions are assessed.
• Figure 2: (a) Multi-branch fusion attention (MFA) module. Key components include hierarchical information fusion attention (HIFA) for diverse local information enhancement and channel-wise local information attention (CLIA) for improved channel-specific representation learning.
• Figure 3: (a) Multimodal information fusion attention (MIFA) module. This module includes multimodal global information fusion attention (MGIFA) (shown in b) and multimodal local information fusion attention (MLIFA) (shown in c).
• Figure 4: Visual representation of the important regions highlighted by our proposed DRIFA-Net and four SOTA methods using the GRAD-CAM technique on two benchmark datasets D1 and D3. (a) and (g) display the original images, while (b) and (h) present results for Gloria, (c) and (i) for MTF with MA, (d) and (j) for CAF, (e) and (k) for MTTU-Net, and (f) and (l) for our proposed DRIFA-Net.
• Figure 5: T-SNE visualization of different models applied to the dermoscopy images of the D1 dataset, where (a) represents the T-SNE visualization of Gloria, (b) of MTTU-Net, and (c) of our proposed DRIFA-Net.