Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization

Yequan Bie, Luyang Luo, Zhixuan Chen, Hao Chen

TL;DR

XCoOp addresses the need for interpretable prompt learning in medical image diagnosis by aligning image semantics, soft prompts, and clinical concept prompts across multiple granularities. It introduces concept-driven prompts built from expert knowledge or elicited via large language models, and couples soft-hard prompt alignment with a global-local image-prompt alignment objective managed by a CLIP backbone. The method achieves superior diagnostic performance across Derm7pt, SkinCon, Pneumonia, and IU X-Ray while providing both textual and visual explanations, demonstrated through faithfulness and plausibility analyses. This work underscores the potential of foundation models to support trusted AI in high-stakes healthcare settings and promises practical impact through improved interpretability and availability of code.

Abstract

Utilizing potent representations of the large vision-language models (VLMs) to accomplish various downstream tasks has attracted increasing attention. Within this research field, soft prompt learning has become a representative approach for efficiently adapting VLMs such as CLIP, to tasks like image classification. However, most existing prompt learning methods learn text tokens that are unexplainable, which cannot satisfy the stringent interpretability requirements of Explainable Artificial Intelligence (XAI) in high-stakes scenarios like healthcare. To address this issue, we propose a novel explainable prompt learning framework that leverages medical knowledge by aligning the semantics of images, learnable prompts, and clinical concept-driven prompts at multiple granularities. Moreover, our framework addresses the lack of valuable concept annotations by eliciting knowledge from large language models and offers both visual and textual explanations for the prompts. Extensive experiments and explainability analyses conducted on various datasets, with and without concept labels, demonstrate that our method simultaneously achieves superior diagnostic performance, flexibility, and interpretability, shedding light on the effectiveness of foundation models in facilitating XAI. The code will be made publically available.

XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization

TL;DR

XCoOp addresses the need for interpretable prompt learning in medical image diagnosis by aligning image semantics, soft prompts, and clinical concept prompts across multiple granularities. It introduces concept-driven prompts built from expert knowledge or elicited via large language models, and couples soft-hard prompt alignment with a global-local image-prompt alignment objective managed by a CLIP backbone. The method achieves superior diagnostic performance across Derm7pt, SkinCon, Pneumonia, and IU X-Ray while providing both textual and visual explanations, demonstrated through faithfulness and plausibility analyses. This work underscores the potential of foundation models to support trusted AI in high-stakes healthcare settings and promises practical impact through improved interpretability and availability of code.

Abstract

Utilizing potent representations of the large vision-language models (VLMs) to accomplish various downstream tasks has attracted increasing attention. Within this research field, soft prompt learning has become a representative approach for efficiently adapting VLMs such as CLIP, to tasks like image classification. However, most existing prompt learning methods learn text tokens that are unexplainable, which cannot satisfy the stringent interpretability requirements of Explainable Artificial Intelligence (XAI) in high-stakes scenarios like healthcare. To address this issue, we propose a novel explainable prompt learning framework that leverages medical knowledge by aligning the semantics of images, learnable prompts, and clinical concept-driven prompts at multiple granularities. Moreover, our framework addresses the lack of valuable concept annotations by eliciting knowledge from large language models and offers both visual and textual explanations for the prompts. Extensive experiments and explainability analyses conducted on various datasets, with and without concept labels, demonstrate that our method simultaneously achieves superior diagnostic performance, flexibility, and interpretability, shedding light on the effectiveness of foundation models in facilitating XAI. The code will be made publically available.
Paper Structure (12 sections, 4 equations, 3 figures, 4 tables)

This paper contains 12 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Clinical Concept-Driven Prompt Design
  4. Soft-Hard Prompt Alignment
  5. Global-Local Image-Prompt Alignment
  6. Experiments
  7. Experimental Setup
  8. Experimental Results
  9. Diagnosis Results.
  10. Ablation Study.
  11. Analysis of Explainability
  12. Conclusion

Figures (3)

  • Figure 1: The overall pipeline of XCoOp. The key insight of XCoOp is enhancing the informativeness and explainability of the soft prompts under the guidance of concept-based medical knowledge at multiple granularities, achieving FM-enhanced XAI.
  • Figure 2: Illustration of our model's faithfulness using knowledge intervention. (a) Clarification of prompt examples based on different knowledge and intervention. (b) The results of concept-based knowledge (Kg) intervention on Derm7pt, where x-axis represents different kinds of prompts and y-axis represents the AUC [%], respectively.
  • Figure 3: Visual explanations. (a) Visualization examples of the similarity between the images and soft prompts. (b) The t-SNE visualization of tokens of different soft prompts of SkinCon (top) and IU X-ray (bottom) datasets. Different colors represent different categories of prompts, and the number of context tokens is 4.