Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis

Townim F. Chowdhury, Vu Minh Hieu Phan, Kewen Liao, Minh-Son To, Yutong Xie, Anton van den Hengel, Johan W. Verjans, Zhibin Liao

TL;DR

Problem: Medical image diagnosis demands transparent reasoning; CLIP-based CBMs offer interpretable concept explanations but struggle with domain transfer due to generic pretraining. Approach: AdaCBM inserts a lightweight adapter between CLIP and CBM to align image embeddings with concept space, while using a fixed concept base and a stationary mask to restrict concept contributions; this yields a linear, interpretable logit for each class without fully retraining the backbone. Contributions: (i) linear reinterpretation of CBMs with an adaptive bridge; (ii) GPT-4–driven, prompt-engineered concept generation for visual attributes and concise phrases; (iii) a concept-utility filter combining Welch's t-test and Pearson correlation to select discriminative, non-redundant concepts. Experiments: Across HAM, BCCD, and DR, AdaCBM achieves accuracy comparable to a linear classifier with the same backbone and outperforms post-CBM fine-tuning, with robust ablations on adapter placement and concept generation. Impact: This framework offers an end-to-end recipe to leverage GPT, CLIP, and CBM in medical diagnostics with strong interpretability and practical training efficiency.

Abstract

The integration of vision-language models such as CLIP and Concept Bottleneck Models (CBMs) offers a promising approach to explaining deep neural network (DNN) decisions using concepts understandable by humans, addressing the black-box concern of DNNs. While CLIP provides both explainability and zero-shot classification capability, its pre-training on generic image and text data may limit its classification accuracy and applicability to medical image diagnostic tasks, creating a transfer learning problem. To maintain explainability and address transfer learning needs, CBM methods commonly design post-processing modules after the bottleneck module. However, this way has been ineffective. This paper takes an unconventional approach by re-examining the CBM framework through the lens of its geometrical representation as a simple linear classification system. The analysis uncovers that post-CBM fine-tuning modules merely rescale and shift the classification outcome of the system, failing to fully leverage the system's learning potential. We introduce an adaptive module strategically positioned between CLIP and CBM to bridge the gap between source and downstream domains. This simple yet effective approach enhances classification performance while preserving the explainability afforded by the framework. Our work offers a comprehensive solution that encompasses the entire process, from concept discovery to model training, providing a holistic recipe for leveraging the strengths of GPT, CLIP, and CBM.

AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis

TL;DR

Problem: Medical image diagnosis demands transparent reasoning; CLIP-based CBMs offer interpretable concept explanations but struggle with domain transfer due to generic pretraining. Approach: AdaCBM inserts a lightweight adapter between CLIP and CBM to align image embeddings with concept space, while using a fixed concept base and a stationary mask to restrict concept contributions; this yields a linear, interpretable logit for each class without fully retraining the backbone. Contributions: (i) linear reinterpretation of CBMs with an adaptive bridge; (ii) GPT-4–driven, prompt-engineered concept generation for visual attributes and concise phrases; (iii) a concept-utility filter combining Welch's t-test and Pearson correlation to select discriminative, non-redundant concepts. Experiments: Across HAM, BCCD, and DR, AdaCBM achieves accuracy comparable to a linear classifier with the same backbone and outperforms post-CBM fine-tuning, with robust ablations on adapter placement and concept generation. Impact: This framework offers an end-to-end recipe to leverage GPT, CLIP, and CBM in medical diagnostics with strong interpretability and practical training efficiency.

Abstract

The integration of vision-language models such as CLIP and Concept Bottleneck Models (CBMs) offers a promising approach to explaining deep neural network (DNN) decisions using concepts understandable by humans, addressing the black-box concern of DNNs. While CLIP provides both explainability and zero-shot classification capability, its pre-training on generic image and text data may limit its classification accuracy and applicability to medical image diagnostic tasks, creating a transfer learning problem. To maintain explainability and address transfer learning needs, CBM methods commonly design post-processing modules after the bottleneck module. However, this way has been ineffective. This paper takes an unconventional approach by re-examining the CBM framework through the lens of its geometrical representation as a simple linear classification system. The analysis uncovers that post-CBM fine-tuning modules merely rescale and shift the classification outcome of the system, failing to fully leverage the system's learning potential. We introduce an adaptive module strategically positioned between CLIP and CBM to bridge the gap between source and downstream domains. This simple yet effective approach enhances classification performance while preserving the explainability afforded by the framework. Our work offers a comprehensive solution that encompasses the entire process, from concept discovery to model training, providing a holistic recipe for leveraging the strengths of GPT, CLIP, and CBM.
Paper Structure (5 sections, 3 equations, 3 figures, 2 tables)

This paper contains 5 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Conclusion
  5. Disclosure of Interests.

Figures (3)

  • Figure 1: A geometrical representation of the CBM framework. The cosine similarity of an image $\mathbf{x}$ and a text concept $\mathbf{t}_j$ is represented by the distance from the point $\hat{\mathbf{x}}$ on the sphere to the classification boundary (black line), i.e., the purple $\hat{\mathbf{x}} \cdot \hat{\mathbf{t}}_j$ quantity. Our adapter module provides adaptation ability to $\mathbf{x}$ as indicated by the green arrows.
  • Figure 2: A comparison of AdaCBM and LaBo generated interpretation with the top-3 contributed concepts where both models predict the correct class. The red and blue colored bars show the concept contribution $\mathbf{W}_{ji} s_j$ and the cosine similarity $\hat{\mathbf{x}} \cdot \hat{\mathbf{t}}$. Clinical-relevant phrases are highlighted in orange color. ✗ indicates a concept that is not initially selected for the predicted class.
  • Figure 3: Training and validation accuracy comparison of LaBo (10K) and our AdaCBM for $k \in \{10, 20, 50\}$ concepts (distinguished by line styles). Due to significant training length differences, we show AdaCBM in orange and LaBo (10k) in blue colored axis.