Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, Yanfeng Wang

TL;DR

A novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection, integrating multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels.

Abstract

Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD

Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

TL;DR

A novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection, integrating multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels.

Abstract

Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD
Paper Structure (16 sections, 6 equations, 18 figures, 11 tables)

This paper contains 16 sections, 6 equations, 18 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Train: Multi-Level Feature Adaptation
  5. Test: Multi-Level Feature Comparison
  6. Experiments
  7. Experimental Setups
  8. Comparison with State-of-the-art Methods
  9. Ablation Studies
  10. Visualization Analysis
  11. Conclusion
  12. Medical Anomaly Detection Benchmark
  13. Text Prompt Formatting
  14. Additional Quantitative Results
  15. Additional Qualitative Results
  16. ...and 1 more sections

Figures (18)

  • Figure 1: The overview of adaptation in pre-trained visual-language models for zero-/few-shot medical anomaly classification (AC) and anomaly segmentation (AS).
  • Figure 2: The architecture of multi-level adaptation and comparison framework for zero-/few-shot medical anomaly detection.
  • Figure 4: Considering features across multiple levels significantly enhances segmentation performance. The white dashed boxes demarcate regions that have been missed or erroneously segmented. More cases are included in Appendix \ref{['sec:ap_vis']}.
  • Figure 5: Visualization of AS for MVFA on brain MRI, liver CT, and retinal OCT, compared with state-of-the-art method April-GAN. Results from (e) show better performance than results from (c), showing the effectiveness of the proposed multi-level feature adaptation.
  • Figure 7: Lists of state and template level prompts employed in this paper to construct text features.
  • ...and 13 more figures