Quasi-multimodal-based pathophysiological feature learning for retinal disease diagnosis

TL;DR

This paper introduces a quasi-multimodal framework for retinal disease diagnosis that synthesizes arterial- and arteriovenous-phase FFA, MSI, and saliency maps from CFP images and fuses them via a two-stage process. Stage I learns modality-specific representations through parallel encoders, while Stage II uses MFFM and CPAM to finetune and adaptively fuse features for disease classification and DR grading. The Cross-Pathophysiology Attention Module enables hierarchical, task-driven calibration across modalities, improving accuracy, robustness, and interpretability. Experiments on MuReD and DDR demonstrate state-of-the-art performance, with additional analyses of modality contributions, visualization of learned features, and resilience to common image perturbations, supporting the framework’s potential for scalable, data-efficient retinal screening. Overall, the work advances a unified, data-efficient approach to multimodal retinal diagnosis with strong clinical relevance and generalizability prospects.

Abstract

Retinal diseases spanning a broad spectrum can be effectively identified and diagnosed using complementary signals from multimodal data. However, multimodal diagnosis in ophthalmic practice is typically challenged in terms of data heterogeneity, potential invasiveness, registration complexity, and so on. As such, a unified framework that integrates multimodal data synthesis and fusion is proposed for retinal disease classification and grading. Specifically, the synthesized multimodal data incorporates fundus fluorescein angiography (FFA), multispectral imaging (MSI), and saliency maps that emphasize latent lesions as well as optic disc/cup regions. Parallel models are independently trained to learn modality-specific representations that capture cross-pathophysiological signatures. These features are then adaptively calibrated within and across modalities to perform information pruning and flexible integration according to downstream tasks. The proposed learning system is thoroughly interpreted through visualizations in both image and feature spaces. Extensive experiments on two public datasets demonstrated the superiority of our approach over state-of-the-art ones in the tasks of multi-label classification (F1-score: 0.683, AUC: 0.953) and diabetic retinopathy grading (Accuracy:0.842, Kappa: 0.861). This work not only enhances the accuracy and efficiency of retinal disease screening but also offers a scalable framework for data augmentation across various medical imaging modalities.

Figures (20)

• Figure 1: Overview of the study: (a) data collection and preprocessing, (b) two-stage implementation of the modelling strategy, and (c) evaluation and interpretation.
• Figure 2: Illustration of modality-specific representation learning. Parallel models, uniformly backboned on an encoder-decoder structure, are utilized to map CFP to multiple modalities across the pathophysiological spectrum. Representative losses used to regularize multimodal images synthesis are indicated, including adversarial loss ($\mathcal{L}_{adv}$) and cycle-consistency loss ($\mathcal{L}_{cyc})$
• Figure 3: Architecture of the multimodal diagnostic system. The system consists of a Multimodal Feature Finetuning Module (MFFM) followed by a Cross-Pathophysiology Attention Module (CPAM). Multimodal features generated by the encoders are first refined by the MFFM, which jointly finetunes these encoders along with the overall diagnostic pipeline to align image synthesis with diagnostic objectives. The refined features are then further calibrated by the CPAM, which employs a hierarchically cascaded attention mechanism to enable fine-grained, multi-scale alignment of pathophysiological features across modalities.
• Figure 4: Architecture of the k-th cross-attention unit in the phase of cross-modal feature calibration. Norm represents layer normalization,$\oplus$ and $\otimes$ represents matrix addition and matrix multiplication, respectively.
• Figure 5: Histograms of disease and DR grade in the datasets of (a) MuReD and (b) DDR, respectively.