Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?

TL;DR

Evaluating CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset highlights the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.

Abstract

Zero-shot anomaly detection (ZSAD) offers potential for identifying anomalies in medical imaging without task-specific training. In this paper, we evaluate CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset. Our analysis examines their ability to detect medical-specific anomalies with no or minimal supervision, addressing the challenges posed by limited data annotation. While these models show promise in transferring general knowledge to medical tasks, their performance falls short of the precision required for clinical use. Our findings highlight the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.

Figures (5)

• Figure 1: Distribution of 3D Dice scores across subjects grouped by AD method and for each training setup: industrial, finetune, brats, and pmc.
• Figure 2: Boxplots of 2D Dice Scores for each subject across all models and datasets. Red crosses represent corresponding 3D Dice Scores for each subject. BraTS Subject IDs are listed at the bottom in the same order as they appear in the corresponding bar charts, from left to right.
• Figure 3: Mean 2D Dice Scores plotted against normalized distance from the center of the brain (0). A value of -1 corresponds to the inferior part of the brain, while 1 represents the superior part. Yellow bars indicate the number of slices with lesions at each distance.
• Figure 4: Sample 2D slices from 3D MRI images and their corresponding whole tumor segmentation (yellow overlap) for inferior part of the brain (a, distance $<$ 0) and superior part of the brain (b, distance $>$ 0).
• Figure 5: Overlay of ground truth and model predictions for MRI segmentation, highlighting true positives (green), false positives (red), and false negatives (blue). Grayscale MRI slices provide anatomical context, showcasing model accuracy and errors. All the sample results refer to models finetuned on BraTS; better viewed in color.