Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quadrant Segmentation VLM with Few-Shot Adaptation and OCT Learning-based Explainability Methods for Diabetic Retinopathy

Shivum Telang

TL;DR

This work tackles DR screening by introducing a quadrant-based Vision-Language Model with few-shot adaptation that jointly processes fundus and OCT images. It integrates Grad-CAM heatmaps for OCT and a quadrant-aware Faster R-CNN framework to localize DR lesions, then translates lesion counts and types per quadrant into natural language explanations. The approach demonstrates high lesion-detection sensitivity across key DR lesion types and provides interpretable NL outputs alongside visual heatmaps, addressing the black-box concern in clinical workflows. The method holds promise for scalable, explainable DR screening and could be extended with longitudinal data, federated learning, and diffusion-transformer explainability techniques.

Abstract

Diabetic Retinopathy (DR) is a leading cause of vision loss worldwide, requiring early detection to preserve sight. Limited access to physicians often leaves DR undiagnosed. To address this, AI models utilize lesion segmentation for interpretability; however, manually annotating lesions is impractical for clinicians. Physicians require a model that explains the reasoning for classifications rather than just highlighting lesion locations. Furthermore, current models are one-dimensional, relying on a single imaging modality for explainability and achieving limited effectiveness. In contrast, a quantitative-detection system that identifies individual DR lesions in natural language would overcome these limitations, enabling diverse applications in screening, treatment, and research settings. To address this issue, this paper presents a novel multimodal explainability model utilizing a VLM with few-shot learning, which mimics an ophthalmologist's reasoning by analyzing lesion distributions within retinal quadrants for fundus images. The model generates paired Grad-CAM heatmaps, showcasing individual neuron weights across both OCT and fundus images, which visually highlight the regions contributing to DR severity classification. Using a dataset of 3,000 fundus images and 1,000 OCT images, this innovative methodology addresses key limitations in current DR diagnostics, offering a practical and comprehensive tool for improving patient outcomes.

Quadrant Segmentation VLM with Few-Shot Adaptation and OCT Learning-based Explainability Methods for Diabetic Retinopathy

TL;DR

This work tackles DR screening by introducing a quadrant-based Vision-Language Model with few-shot adaptation that jointly processes fundus and OCT images. It integrates Grad-CAM heatmaps for OCT and a quadrant-aware Faster R-CNN framework to localize DR lesions, then translates lesion counts and types per quadrant into natural language explanations. The approach demonstrates high lesion-detection sensitivity across key DR lesion types and provides interpretable NL outputs alongside visual heatmaps, addressing the black-box concern in clinical workflows. The method holds promise for scalable, explainable DR screening and could be extended with longitudinal data, federated learning, and diffusion-transformer explainability techniques.

Abstract

Diabetic Retinopathy (DR) is a leading cause of vision loss worldwide, requiring early detection to preserve sight. Limited access to physicians often leaves DR undiagnosed. To address this, AI models utilize lesion segmentation for interpretability; however, manually annotating lesions is impractical for clinicians. Physicians require a model that explains the reasoning for classifications rather than just highlighting lesion locations. Furthermore, current models are one-dimensional, relying on a single imaging modality for explainability and achieving limited effectiveness. In contrast, a quantitative-detection system that identifies individual DR lesions in natural language would overcome these limitations, enabling diverse applications in screening, treatment, and research settings. To address this issue, this paper presents a novel multimodal explainability model utilizing a VLM with few-shot learning, which mimics an ophthalmologist's reasoning by analyzing lesion distributions within retinal quadrants for fundus images. The model generates paired Grad-CAM heatmaps, showcasing individual neuron weights across both OCT and fundus images, which visually highlight the regions contributing to DR severity classification. Using a dataset of 3,000 fundus images and 1,000 OCT images, this innovative methodology addresses key limitations in current DR diagnostics, offering a practical and comprehensive tool for improving patient outcomes.
Paper Structure (8 sections, 3 equations, 5 figures, 1 table)

This paper contains 8 sections, 3 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Data Curation
  5. Grad-CAM Explainability Framework
  6. Quadrant Segmentation VLM with Few-Shot Learning
  7. Results
  8. Conclusion and Future Work

Figures (5)

  • Figure 1: The image to the left shows a fundus scan (picture of retina) and the image to the right shows an OCT scan (structure of retina)
  • Figure 2: Grad-CAM Heatmap Architecture
  • Figure 3: VLM with Few-Shot Learning Explainability Filter
  • Figure 4: Fundus Image Quadrant-based Lesion Extraction Response
  • Figure 5: Grad-CAM Heatmap overlay for different classes