M3T: Multi-Modal Medical Transformer to bridge Clinical Context with Visual Insights for Retinal Image Medical Description Generation

TL;DR

The paper introduces M3T, a Multi-Modal Medical Transformer designed to generate precise medical descriptions from retinal images by fusing visual cues with diagnostic keywords. It comprises a Visual Encoder (with a Convolutional Base and Lesion Contextual Gate), a Keywords Encoder, a TransFusion Encoder to integrate modalities, and a Medical Description Decoder based on Transformer decoding. Quantitative results on the DeepEyeNet dataset show M3T outperforming prior models, marked by a $BLEU@4$ improvement of 13.5% over the best baseline, while qualitative analyses provide interpretable attention heatmaps illustrating which regions and keywords drive descriptions. This approach advances automated ophthalmic reporting, enabling more context-rich and clinically relevant narratives, and can be extended to broader medical imaging domains with improved interpretability.

Abstract

Automated retinal image medical description generation is crucial for streamlining medical diagnosis and treatment planning. Existing challenges include the reliance on learned retinal image representations, difficulties in handling multiple imaging modalities, and the lack of clinical context in visual representations. Addressing these issues, we propose the Multi-Modal Medical Transformer (M3T), a novel deep learning architecture that integrates visual representations with diagnostic keywords. Unlike previous studies focusing on specific aspects, our approach efficiently learns contextual information and semantics from both modalities, enabling the generation of precise and coherent medical descriptions for retinal images. Experimental studies on the DeepEyeNet dataset validate the success of M3T in meeting ophthalmologists' standards, demonstrating a substantial 13.5% improvement in BLEU@4 over the best-performing baseline model.

Figures (2)

• Figure 1: Architecture of proposed Multi-Modal Medical Transformer (M3T); Visual Encoder -- learns attention-based representations from retinal images; Keyword Encoder -- learns clinical-context embeddings from diagnostic keywords; TransFusion Encoder -- integrates visual attention features and clinical-context embeddings, leveraging both visual and semantic information; Medical Description Generation Decoder -- generates coherent and meaningful medical descriptions by attending to relevant visual and semantic cues, ensuring contextually appropriate and diagnostically relevant outputs;
• Figure 2: Visualization of Lesion Contextual Gate Attention heatmaps and corresponding overlays, highlighting the visual insights from color fundus, gray fundus, OCT, and Auto-Fluorescent retinal scan images