CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification

TL;DR

CBVLM introduces a training-free, two-stage framework that grounds medical image classification in LVLM-predicted concepts, addressing annotation burden and interpretability. By combining concept detection with concept-grounded disease diagnosis and leveraging retrieval-based few-shot prompts, CBVLM achieves competitive or superior performance to CBMs and supervised models across four medical datasets while using minimal annotated data. The approach demonstrates robust gains from few-shot prompting, emphasizes explainability through concept grounding, and remains adaptable to adding new concepts without retraining. Limitations include scalability to many concepts and reliance on LVLM reliability, suggesting avenues for improved concept selection and larger-model investigations.

Abstract

The main challenges limiting the adoption of deep learning-based solutions in medical workflows are the availability of annotated data and the lack of interpretability of such systems. Concept Bottleneck Models (CBMs) tackle the latter by constraining the model output on a set of predefined and human-interpretable concepts. However, the increased interpretability achieved through these concept-based explanations implies a higher annotation burden. Moreover, if a new concept needs to be added, the whole system needs to be retrained. Inspired by the remarkable performance shown by Large Vision-Language Models (LVLMs) in few-shot settings, we propose a simple, yet effective, methodology, CBVLM, which tackles both of the aforementioned challenges. First, for each concept, we prompt the LVLM to answer if the concept is present in the input image. Then, we ask the LVLM to classify the image based on the previous concept predictions. Moreover, in both stages, we incorporate a retrieval module responsible for selecting the best examples for in-context learning. By grounding the final diagnosis on the predicted concepts, we ensure explainability, and by leveraging the few-shot capabilities of LVLMs, we drastically lower the annotation cost. We validate our approach with extensive experiments across four medical datasets and twelve LVLMs (both generic and medical) and show that CBVLM consistently outperforms CBMs and task-specific supervised methods without requiring any training and using just a few annotated examples. More information on our project page: https://cristianopatricio.github.io/CBVLM/.

Figures (22)

• Figure 1: Comparison between the proposed methodology (CBVLM) with existing methods. Unlike traditional approaches, our methodology grounds the final diagnosis on a set of clinical concepts predicted by the LVLM itself (cf.Concept Detector), thus increasing classification performance, while being interpretable.
• Figure 2: Overview of CBVLM. Our methodology is organized into two key stages: 1) The Concept Detection stage, where the LVLM predicts the individual presence of each predefined clinical concept in the query image. This is achieved using a custom prompt that supports both zero- and few-shot settings. In the latter, we include a set of demonstration examples (middle block of Prompt Construction) chosen by the Retrieval Module, responsible for selecting the $N$ most similar examples to the input image. To evaluate the LVLM answer, we employ an Evaluation Block which first tries to extract the desired LVLM response using a rule-based formulation. If this fails, we adopt an auxiliary LLM to extract the desired response. 2) In the Disease Diagnosis stage, the final diagnosis is generated by the LVLM based on the clinical concepts predicted in the first stage, which are directly incorporated in the query (highlighted in yellow). This approach ensures that the diagnosis is grounded on the identified clinical concepts, enhancing the interpretability and transparency of the LVLM’s response. In this second stage, the Retrieval Module is also used to select the $N$ most similar demonstrations.
• Figure 3: Concept detection performance of LVLMs across different $n$-shot settings. Each bar corresponds to an $n$-shot scenario ($n = \{0,1,2,4\}$). Filled colored bars denote BACC, whereas the hatched bars indicate F1-scores. Red crosses indicate the percentage of unknowns, i.e. the proportion of samples whose LVLMs' responses do not contain sufficient information to answer the posed question.
• Figure 4: Concept detection results per dataset averaged over all models (left) and over generic and medical LVLMs (right). Each bar corresponds to the number of shots ($n = \{0,1,2,4\}$). Filled colored bars denote BACC, whereas the hatched bars indicate F1-scores.
• Figure 5: Per-concept F1-scores across datasets averaged over all (blue), generic (green), and medical (orange) LVLMs, in the 4-shot scenario. Values in parentheses indicate the number of samples in the dataset for the corresponding concept.