Confidence-aware multi-modality learning for eye disease screening

TL;DR

This work addresses the need for calibrated confidence and robustness in multimodal eye disease screening by fusing Fundus and OCT data. It introduces EyeMoS$t+$, which models per-modality uncertainty through $NIG$ priors and analytically maps them to predictive $t$-distributions, subsequently fused via a confidence-aware Mixture of $t$ distributions ($MoS$t$) with a ranking regularization to keep fusion confidence higher than any single modality. The framework delivers improved reliability across glaucoma, AMD/PCV, DR/DME datasets and demonstrates robustness to Gaussian noise, missing modalities, and out-of-distribution samples. Its principled uncertainty-aware fusion is poised to enhance clinical safety and generalization in real-world multimodal ophthalmic screening.

Abstract

Multi-modal ophthalmic image classification plays a key role in diagnosing eye diseases, as it integrates information from different sources to complement their respective performances. However, recent improvements have mainly focused on accuracy, often neglecting the importance of confidence and robustness in predictions for diverse modalities. In this study, we propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information using a multi-distribution fusion perspective. Specifically, our method first utilizes normal inverse gamma prior distributions over pre-trained models to learn both aleatoric and epistemic uncertainty for uni-modality. Then, the normal inverse gamma distribution is analyzed as the Student's t distribution. Furthermore, within a confidence-aware fusion framework, we propose a mixture of Student's t distributions to effectively integrate different modalities, imparting the model with heavy-tailed properties and enhancing its robustness and reliability. More importantly, the confidence-aware multi-modality ranking regularization term induces the model to more reasonably rank the noisy single-modal and fused-modal confidence, leading to improved reliability and accuracy. Experimental results on both public and internal datasets demonstrate that our model excels in robustness, particularly in challenging scenarios involving Gaussian noise and modality missing conditions. Moreover, our model exhibits strong generalization capabilities to out-of-distribution data, underscoring its potential as a promising solution for multimodal eye disease screening.

Figures (7)

• Figure 1: Comparison multi-modality classification methods for eye disease screening. (a) Traditional multi-modality eye disease screening. (b) Our confidence-aware multi-modality learning for eye disease screening. $\hat{y}$ and $U$ denote the prediction and its uncertainty, respectively.
• Figure 2: The framework of confidence-aware multi-modality learning for eye disease screening (EyeMoS$t+$).
• Figure 3: Confidence-aware fusion for Mixture of Student's $t$ distributions and Confidence-aware multi-modality ranking.
• Figure 4: Accuracy and ECE performance of different algorithms in noisy single modality with different levels of noise on GAMMA dataset. (a) ACC and ECE for various algorithms in the presence of noise at different levels in Fundus modality. (b) ACC and ECE for various algorithms in the presence of noise at different levels in OCT modality. Higher ACC and Lower ECE mean better.
• Figure 5: Results from Robust validation. (a-b) Performance metrics including ACC, Kappa, AURC, and ECE for various algorithms in the presence of noise at different levels in single modalities. Results are shown for both OLIVES and in-house datasets. (c) Comparisons of original and noisy OCT/Fundus data on the In-house dataset.