Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models

TL;DR

The paper tackles the lack of interpretability and data scarcity in retinal disease diagnosis from color fundus images by introducing a two-stage framework that first builds a GPT-derived concept bank validated by ophthalmologists and then trains vision-language models to predict these concepts. A subsequent concept bottleneck classifier maps predicted concepts to disease categories, enabling interpretable disease diagnosis. Across in-house and RFMiD datasets, the approach delivers notable gains in few-shot (≈5.8% mAP) and zero-shot (≈2.7% mAP) performance and provides transparent concept-to-disease explanations via learned concept contributions. This work advances interpretable, data-efficient retinal disease recognition with potential for real-world clinical deployment and extension to other medical imaging domains.

Abstract

Recent advancements in deep learning have shown significant potential for classifying retinal diseases using color fundus images. However, existing works predominantly rely exclusively on image data, lack interpretability in their diagnostic decisions, and treat medical professionals primarily as annotators for ground truth labeling. To fill this gap, we implement two key strategies: extracting interpretable concepts of retinal diseases using the knowledge base of GPT models and incorporating these concepts as a language component in prompt-learning to train vision-language (VL) models with both fundus images and their associated concepts. Our method not only improves retinal disease classification but also enriches few-shot and zero-shot detection (novel disease detection), while offering the added benefit of concept-based model interpretability. Our extensive evaluation across two diverse retinal fundus image datasets illustrates substantial performance gains in VL-model based few-shot methodologies through our concept integration approach, demonstrating an average improvement of approximately 5.8\% and 2.7\% mean average precision for 16-shot learning and zero-shot (novel class) detection respectively. Our method marks a pivotal step towards interpretable and efficient retinal disease recognition for real-world clinical applications.

Figures (4)

• Figure 1: Our framework consists of two stages: Stage 1 focuses on developing a concept bank through GPT and validation by ophthalmologists, followed by concept-guided prompt learning of VL models for concept prediction. Stage 2 trains machine learning models to classify disease categories based on Stage 1's concepts.
• Figure 2: Design of two prompt templates for constructing the retinal disease concept bank. The figure illustrates the prompts for the retinal disease of Asteroid Hyalosis with adaptable disease-specific parts. These two templates prompt LLMs twice, generating concepts that are later validated by ophthalmologists.
• Figure 3: Fundus images illustrating key representative attributes of Diabetic Retinopathy (DR) and Central Retinal Vein Occlusion (CRVO) and a sankey diagram depicting the flow of the concepts relevant to DR and CRVO learnt by our framework. We present the top 5 and the bottom 5 concepts associated with each DR and CRVO based on the contribution scores of the concepts to DR and CRVO from our LR model. The scores are averaged and normalized over all the test samples of DR and CRVO.
• Figure 4: Fundus image illustrating key representative attributes of Choroidal hemangioma and a sankey diagram depicting the flow of the concepts relevant to the novel category of Choroidal hemangioma in zero-shot detection setting of our framework. We present the top 4 and the bottom 4 concepts associated with choroidal hemangioma learnt by our model. The scores are averaged and normalized over all the test samples choroidal hemangioma.