Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis

TL;DR

The paper tackles the poor cross-modal alignment of general vision-language models in ophthalmology by introducing a knowledge-enhanced multimodal transformer that jointly embeds retinal images, clinical text, and structured patient data. Each modality is processed by a specialized encoder (ViT for images, Bio-ClinicalBERT for text, and an MLP for structured data) and fused via a modality-aware transformer, trained with a balanced set of contrastive, reconstruction, and supervised losses. On BRSET, the approach delivers near-perfect text-to-image retrieval (Recall@1 = 99.94%) while maintaining strong DR severity classification (SDRG 97.05%, ICDR 97.97%), and demonstrates robust zero-shot generalization to an unseen dataset (Recall@1 = 93.95%). These results highlight the value of domain-specific, multi-modal representations for clinical decision support and point toward practical deployment in diverse clinical settings, including resource-limited environments.

Abstract

Diabetic retinopathy (DR) is a leading cause of preventable blindness worldwide, demanding accurate automated diagnostic systems. While general-domain vision-language models like Contrastive Language-Image Pre-Training (CLIP) perform well on natural image tasks, they struggle in medical domain applications, particularly in cross-modal retrieval for ophthalmological images. We propose a novel knowledge-enhanced joint embedding framework that integrates retinal fundus images, clinical text, and structured patient data through a multimodal transformer architecture to address the critical gap in medical image-text alignment. Our approach employs separate encoders for each modality: a Vision Transformer (ViT-B/16) for retinal images, Bio-ClinicalBERT for clinical narratives, and a multilayer perceptron for structured demographic and clinical features. These modalities are fused through a joint transformer with modality-specific embeddings, trained using multiple objectives including contrastive losses between modality pairs, reconstruction losses for images and text, and classification losses for DR severity grading according to ICDR and SDRG schemes. Experimental results on the Brazilian Multilabel Ophthalmological Dataset (BRSET) demonstrate significant improvements over baseline models. Our framework achieves near-perfect text-to-image retrieval performance with Recall@1 of 99.94% compared to fine-tuned CLIP's 1.29%, while maintaining state-of-the-art classification accuracy of 97.05% for SDRG and 97.97% for ICDR. Furthermore, zero-shot evaluation on the unseen DeepEyeNet dataset validates strong generalizability with 93.95% Recall@1 versus 0.22% for fine-tuned CLIP. These results demonstrate that our multimodal training approach effectively captures cross-modal relationships in the medical domain, establishing both superior retrieval capabilities and robust diagnostic performance.

Figures (14)

• Figure 1: Distribution of DR grades in the BRSET training dataset.
• Figure 2: Prevalence of pathological conditions in BRSET dataset beyond diabetic retinopathy, including macular edema, age-related macular degeneration, hypertensive retinopathy, and other retinal conditions.
• Figure 3: Distribution of synthetic clinical note lengths (in tokens) after generation from binary disease indicators. Each note falls well within the 128-token limit, validating the truncation strategy.
• Figure 4: Overview of the ViT-B/16 architecture, followed by a fully connected layer, applied on 16 x 16 retinal image patches.
• Figure 5: Overview of the multimodal fusion transformer encoder comprising six layers, eight attention heads, and a model dimension of 256, enabling cross-modal attention flow.