Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee

Abstract

Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention(TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP

MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

Abstract

Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention(TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP
Paper Structure (24 sections, 16 equations, 4 figures, 7 tables)

This paper contains 24 sections, 16 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Anomaly Detection
  4. Medical Vision Language Model
  5. Margin Contrastive loss
  6. Methodology
  7. Problem Formulation
  8. Overall Architecture
  9. Multi-level Image Adapters and Learnable Prompt
  10. Token-Patch Cross-Attention for Segmentation
  11. Classification
  12. Loss function
  13. Experiments
  14. Datasets and Metrics
  15. Experimental Setup
  16. ...and 9 more sections

Figures (4)

  • Figure 1: (a) Qualitative result comparisons of few-shot MVFA and our supervised MedSAD-CLIP on a Breast image. (b) Radar comparison of MedSAD-CLIP and zero-/few-shot CLIP-based SOTAs on Brain and Breast datasets.
  • Figure 2: Overall architecture of our proposed MedSAD-CLIP. It consists of two branches: the classification branch (Det) and the segmentation branch (Seg). The model extends CLIP radford2021learning with adapter modules and learnable prompts to adapt to medical image analysis.
  • Figure 3: Qualitative comparison between our model MedSAD-CLIP and MVFA. Our MedSAD-CLIP produces sharper and more accurate segmentations.
  • Figure 4: Qualitative results comparing MVFA (few/full) and our method on brain anomaly segmentation, showing both binary masks and score maps.