Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explaining Chest X-ray Pathology Models using Textual Concepts

Vijay Sadashivaiah, Pingkun Yan, James A. Hendler

TL;DR

This paper proposes Conceptual Counterfactual Explanations for Chest X-ray (CoCoX), which leverages the joint embedding space of an existing vision-language model (VLM) to explain black-box classifier outcomes without the need for annotated datasets.

Abstract

Deep learning models have revolutionized medical imaging and diagnostics, yet their opaque nature poses challenges for clinical adoption and trust. Amongst approaches to improve model interpretability, concept-based explanations aim to provide concise and human-understandable explanations of any arbitrary classifier. However, such methods usually require a large amount of manually collected data with concept annotation, which is often scarce in the medical domain. In this paper, we propose Conceptual Counterfactual Explanations for Chest X-ray (CoCoX), which leverages the joint embedding space of an existing vision-language model (VLM) to explain black-box classifier outcomes without the need for annotated datasets. Specifically, we utilize textual concepts derived from chest radiography reports and a pre-trained chest radiography-based VLM to explain three common cardiothoracic pathologies. We demonstrate that the explanations generated by our method are semantically meaningful and faithful to underlying pathologies.

Explaining Chest X-ray Pathology Models using Textual Concepts

TL;DR

This paper proposes Conceptual Counterfactual Explanations for Chest X-ray (CoCoX), which leverages the joint embedding space of an existing vision-language model (VLM) to explain black-box classifier outcomes without the need for annotated datasets.

Abstract

Deep learning models have revolutionized medical imaging and diagnostics, yet their opaque nature poses challenges for clinical adoption and trust. Amongst approaches to improve model interpretability, concept-based explanations aim to provide concise and human-understandable explanations of any arbitrary classifier. However, such methods usually require a large amount of manually collected data with concept annotation, which is often scarce in the medical domain. In this paper, we propose Conceptual Counterfactual Explanations for Chest X-ray (CoCoX), which leverages the joint embedding space of an existing vision-language model (VLM) to explain black-box classifier outcomes without the need for annotated datasets. Specifically, we utilize textual concepts derived from chest radiography reports and a pre-trained chest radiography-based VLM to explain three common cardiothoracic pathologies. We demonstrate that the explanations generated by our method are semantically meaningful and faithful to underlying pathologies.
Paper Structure (10 sections, 4 equations, 4 figures, 2 tables)

This paper contains 10 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Constructing concept bank
  4. Learning projection functions
  5. Learning conceptual perturbations
  6. Experiments
  7. Datasets
  8. Implementation details
  9. Results and Evaluation
  10. Conclusions

Figures (4)

  • Figure 1: The overall architecture of CoCoX. We create the Concept Bank ($C$) by encoding natural language concepts with a chest radiography pre-trained language encoder of a VLM. We then learn Projector(in) and Projector(out) functions $p_{in}$ and $p_{out}$ using MLPs to transfer feature representation between CLF latent space and VLM latent space. Additionally, we learn perturbation weights for each concept in our concept bank, which serves as an importance score in changing the target classifier output. Learnable parameters are highlighted in red arrows.
  • Figure 2: Training paradigm for projection functions. Each $\leftrightarrow$ indicates a distance minimization problem. Total loss is a sum of all loss functions, $\mathcal{L}_{total} = \mathcal{L}_{{in}} + \mathcal{L}_{{out}} + \mathcal{L}_{{cyc}}$.
  • Figure 3: Conceptual counterfactual generated by CoCoX. The input image into the model for each pathology is shown under the heading Input ("No Finding"). The target image presented under Target ("Pathology") is a representative image labeled positive for the corresponding pathology. The concept importance scores for the top 5 concepts are visualized with blue-green for the DenseNet-121 model and orange-purple for VLM + Linear. These concepts had the most impact when added to input images to change from "No Finding" to a pathology finding.
  • Figure S1: Reverse conceptual counterfactual generated by CoCoX. The input image into the model is shown under the heading Input ("Pathology"). The target image presented under Target ("No Finding") is a representative image labeled negative for any pathologies. The concept importance scores for the top 5 concepts are visualized with blue-green for the DenseNet-121 model and orange-purple for VLM + Linear. These concepts had the most impact when subtracted from the input image embedding to change from pathology finding to no finding.