AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection

TL;DR

AnomalyCLIP tackles zero-shot anomaly detection by learning object-agnostic prompts that capture generic normality and abnormality, decoupled from foreground object semantics. It couples global and local context optimization with a Diagonally Prominent Attention Map to align text prompts with both global and fine-grained visual cues, enabling accurate detection and segmentation without target-domain training data. Across 17 real-world datasets spanning industrial and medical domains, it consistently outperforms CLIP-based baselines and demonstrates robust cross-domain generalization, including improvements when fine-tuned with medical data. The approach significantly reduces reliance on extensive prompt engineering and demonstrates strong practical impact for privacy-preserving and deployment-ready anomaly detection systems.

Abstract

Zero-shot anomaly detection (ZSAD) requires detection models trained using auxiliary data to detect anomalies without any training sample in a target dataset. It is a crucial task when training data is not accessible due to various concerns, eg, data privacy, yet it is challenging since the models need to generalize to anomalies across different domains where the appearance of foreground objects, abnormal regions, and background features, such as defects/tumors on different products/organs, can vary significantly. Recently large pre-trained vision-language models (VLMs), such as CLIP, have demonstrated strong zero-shot recognition ability in various vision tasks, including anomaly detection. However, their ZSAD performance is weak since the VLMs focus more on modeling the class semantics of the foreground objects rather than the abnormality/normality in the images. In this paper we introduce a novel approach, namely AnomalyCLIP, to adapt CLIP for accurate ZSAD across different domains. The key insight of AnomalyCLIP is to learn object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its foreground objects. This allows our model to focus on the abnormal image regions rather than the object semantics, enabling generalized normality and abnormality recognition on diverse types of objects. Large-scale experiments on 17 real-world anomaly detection datasets show that AnomalyCLIP achieves superior zero-shot performance of detecting and segmenting anomalies in datasets of highly diverse class semantics from various defect inspection and medical imaging domains. Code will be made available at https://github.com/zqhang/AnomalyCLIP.

Figures (37)

• Figure 1: Comparison of ZSAD results on (b) test data using (c) original text prompts in CLIP radford2021learning, (d) tailored text prompts for AD in WinCLIP jeong2023winclip, (e) learnable text prompts for general vision tasks in CoOp zhou2022conditional, and (f) object-agnostic text prompts in our AnomalyCLIP. (a) presents a set of auxiliary data we can use to learn the text prompts. The results are obtained by measuring the similarity between text prompt embeddings and image embeddings. The ground-truth anomaly regions are circled in red in (a) and (b). (c), (d), and (e) suffer from poor generalization across different domains, while our AnomalyCLIP in (f) can well generalize to anomalies in diverse types of objects from different domains.
• Figure 2: Overview of AnomalyCLIP. To adapt CLIP to ZSAD, AnomalyCLIP introduces object-agnostic text prompt templates to capture generic normality and abnormality regardless of the object semantics. Then, we introduce glocal context optimization to incorporate global and fine-grained anomaly semantics into object-agnostic text prompt learning. Finally, textual prompt tuning and DPAM are used to enable the prompt learning in the textual and local visual spaces of CLIP.
• Figure 3: DPAM visualization.
• Figure 4: Segmentation visualization.
• Figure 4: Module ablation.