Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

Mohamed Harmanani, Bining Long, Zhuoxin Guo, Paul F. R. Wilson, Amirhossein Sabour, Minh Nguyen Nhat To, Gabor Fichtinger, Purang Abolmaesumi, Parvin Mousavi

TL;DR

This work proposes MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models, and forms an end-to-end bridge from medical image analysis to decision-making that enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.

Abstract

Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features to a set of meaningful concepts for task-specific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, discrete concept representations often overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multitask objective combining multimodal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. MedCBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments on non-medical datasets achieve 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.

Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

TL;DR

This work proposes MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models, and forms an end-to-end bridge from medical image analysis to decision-making that enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.

Abstract

Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features to a set of meaningful concepts for task-specific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, discrete concept representations often overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multitask objective combining multimodal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. MedCBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments on non-medical datasets achieve 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.
Paper Structure (11 sections, 4 equations, 11 figures, 6 tables)

This paper contains 11 sections, 4 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Problem Formulation
  5. Guideline-Driven Concept Enrichment
  6. Vision--Language Concept Modelling
  7. Concept-Based Clinical Reasoning
  8. Experiments
  9. Setup
  10. Results
  11. Conclusion

Figures (11)

  • Figure 1: MedCBR: We frame interpretable medical image analysis as reasoning over diverse sources of evidence, including model predictions and clinical guidelines.
  • Figure 2: Overview of the MedCBR framework. MedCBR integrates clinical knowledge, vision--language alignment, and reasoning for interpretable cancer detection. (a) A large vision--language model (LVLM) generates guideline-conformant reports from input images and coarse concept annotations. (b) A concept-based CLIP model aligns visual and textual embeddings while jointly predicting concepts and diagnostic labels. (c) A frozen large reasoning model (LRM) synthesizes guideline information, predicted concepts, and model outputs to produce structured clinical explanations.
  • Figure 3: Quantitative results on concept detection for ultrasound and mammography. For each dataset, we focus on a subset of concepts most relevant to cancer detection.
  • Figure 4: Case study of MedCBR's reasoning on medical images. Clinical concepts highlighted in green support a benign conclusion, those in red support cancer, and those in yellow are neutral. Key observations are emphasized in green (benign) and red (cancer).
  • Figure 5: Validation of MedCBR's reasoning on CUB-200. Observations matching the guideline are highlighted in blue , conflicting ones are highlighted in orange .
  • ...and 6 more figures