GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning

TL;DR

The paper tackles automated retinal image captioning under limited labeled data by proposing GCS-M3VLT, a Vision-Language Transformer that guides self-attention with both spatial and channel context for multi-modal fusion. It introduces a Vision Encoder with Guided Context Attention, a Language Encoder for diagnostic keywords, and a Vision-Language TransFusion Encoder followed by a GPT-2–style decoder. On the DeepEyeNet dataset, the method yields state-of-the-art results, including a BLEU@4 improvement of 0.023 and strong CIDEr/ROUGE scores, while maintaining a lighter parameter footprint. The approach also provides qualitative evidence of lesion-context localization via Grad-CAM and cohesive clinical captions, suggesting practical benefits for automated retinal reporting.

Abstract

Retinal image analysis is crucial for diagnosing and treating eye diseases, yet generating accurate medical reports from images remains challenging due to variability in image quality and pathology, especially with limited labeled data. Previous Transformer-based models struggled to integrate visual and textual information under limited supervision. In response, we propose a novel vision-language model for retinal image captioning that combines visual and textual features through a guided context self-attention mechanism. This approach captures both intricate details and the global clinical context, even in data-scarce scenarios. Extensive experiments on the DeepEyeNet dataset demonstrate a 0.023 BLEU@4 improvement, along with significant qualitative advancements, highlighting the effectiveness of our model in generating comprehensive medical captions.

Figures (4)

• Figure 1: Architecture of the proposed Guided Context Self-Attention-based Multi-modal Medical Vision Language Transformer (GCS-M3VLT); Vision Encoder -- Learns attention-based representations from retinal images to capture visual features crucial for diagnosis; Language Encoder -- Learns self-attention-based clinical-context embeddings from diagnostic keywords, enabling the model to understand the semantic context of medical terms; Vision-Language TransFusion Encoder -- Integrates visual attention features and clinical-context embeddings, leveraging both visual and semantic information to provide comprehensive understanding; Language Generation Decoder -- Generates coherent and meaningful medical descriptions by attending to relevant visual and semantic cues, ensuring contextually appropriate and diagnostically relevant outputs.
• Figure 2: Architecture of Guided Context Attention that utilizes context features to compute lesion contextual attention representations; Spatial Context Formulation -- selectively focuses on relevant spatial features in the initial representations and computes global context information; Channel Context Formulation -- processes the computed context information & capture channel-wise correlations;
• Figure 3: Comparison of Actual and Predicted Captions obtained using the proposed GCS-M3VLT with Existing Works shaik2024gatedhuang2022non using Retinal Image and Keywords as input. The captions generated by GCS-M3VLT closely resemble the ground truth captions, showcasing the model's ability to accurately describe the clinical context of retinal images. In contrast, captions from existing works exhibit discrepancies and lack coherence compared to ground truth, underscoring the superior performance of the proposed approach.
• Figure 4: Visualizing heatmaps and gradient class activation maps obtained using Vision Encoder with Guided Context Attention features of (a) OCT (b & c) fundus + AF (d) OCT + fundus + AF retinal images highlighting lesion context information