Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification

Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang, Zhiyang He, Xiaodong Tao, S. Kevin Zhou

TL;DR

This work introduces a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero), which innovatively leverages cross-attention mechanisms to process image and report features, creating a Similarity Representation that more accurately reflects the intricate relationships in medical semantics.

Abstract

The advancement of Zero-Shot Learning in the medical domain has been driven forward by using pre-trained models on large-scale image-text pairs, focusing on image-text alignment. However, existing methods primarily rely on cosine similarity for alignment, which may not fully capture the complex relationship between medical images and reports. To address this gap, we introduce a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero). Our approach innovatively leverages cross-attention mechanisms to process image and report features, creating a Similarity Representation that more accurately reflects the intricate relationships in medical semantics. This representation is then linearly projected to form an image-text similarity matrix for cross-modality alignment. Additionally, recognizing the pivotal role of prompt selection in zero-shot learning, CARZero incorporates a Large Language Model-based prompt alignment strategy. This strategy standardizes diverse diagnostic expressions into a unified format for both training and inference phases, overcoming the challenges of manual prompt design. Our approach is simple yet effective, demonstrating state-of-the-art performance in zero-shot classification on five official chest radiograph diagnostic test sets, including remarkable results on datasets with long-tail distributions of rare diseases. This achievement is attributed to our new image-text alignment strategy, which effectively addresses the complex relationship between medical images and reports. Code and models are available at https://github.com/laihaoran/CARZero.

CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification

TL;DR

This work introduces a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero), which innovatively leverages cross-attention mechanisms to process image and report features, creating a Similarity Representation that more accurately reflects the intricate relationships in medical semantics.

Abstract

The advancement of Zero-Shot Learning in the medical domain has been driven forward by using pre-trained models on large-scale image-text pairs, focusing on image-text alignment. However, existing methods primarily rely on cosine similarity for alignment, which may not fully capture the complex relationship between medical images and reports. To address this gap, we introduce a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero). Our approach innovatively leverages cross-attention mechanisms to process image and report features, creating a Similarity Representation that more accurately reflects the intricate relationships in medical semantics. This representation is then linearly projected to form an image-text similarity matrix for cross-modality alignment. Additionally, recognizing the pivotal role of prompt selection in zero-shot learning, CARZero incorporates a Large Language Model-based prompt alignment strategy. This strategy standardizes diverse diagnostic expressions into a unified format for both training and inference phases, overcoming the challenges of manual prompt design. Our approach is simple yet effective, demonstrating state-of-the-art performance in zero-shot classification on five official chest radiograph diagnostic test sets, including remarkable results on datasets with long-tail distributions of rare diseases. This achievement is attributed to our new image-text alignment strategy, which effectively addresses the complex relationship between medical images and reports. Code and models are available at https://github.com/laihaoran/CARZero.
Paper Structure (19 sections, 9 equations, 3 figures, 6 tables)

This paper contains 19 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-shot classification
  4. Cross-Attention in Modality Alignment
  5. Prompt Alignment
  6. Method
  7. Feature Extraction
  8. Cross-Attention Alignment
  9. LLM-based Prompt Alignment
  10. Experiments
  11. Dataset
  12. Evaluation Metric
  13. Implementation Details
  14. Comparison with State-of-the-art Methods
  15. Visualization
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Comparison of the alignment scheme in Visual Language Pre-training: (left) handcrafted cosine similarity used in CLIP conde2021clip and CheXzero tiu2022expert; (right) our proposed cross-attention alignment leveraging a novel similarity representation.
  • Figure 2: The CARZero Network proposed in this paper consists of two stages. First, LLM is employed to generate prompt templates from medical reports. Second, text and vision encoders are used to extract features from image and text, which are fed into a cross-attention module to generate similarity for optimizing InfoNCE loss.
  • Figure 3: Visualization of attention map in CARZero on ChestXDet10. The red boxes indicate the corresponding ground truth of detection. Highlighted pixels represent higher activation weights correlating specific words with regions in the image.