AdaptCLIP: Adapting CLIP for Universal Visual Anomaly Detection

TL;DR

AdaptCLIP tackles universal visual anomaly detection across unseen domains without test-domain fine-tuning by introducing three lightweight adapters (visual, textual, and prompt-query) to a CLIP backbone. The method hinges on alternating learning between visual and textual representations and on a comparative learning mechanism that fuses contextual information with aligned residual features between query and normal prompts. Training uses standard segmentation and classification losses, while inference combines zero-/few-shot predictions from all adapters, achieving state-of-the-art results on 12 industrial and medical benchmarks with strong cross-domain generalization and competitive efficiency. This approach preserves CLIP's original capabilities while enabling rapid, training-free adaptation to new domains, making it practical for open-world anomaly detection scenarios.

Abstract

Universal visual anomaly detection aims to identify anomalies from novel or unseen vision domains without additional fine-tuning, which is critical in open scenarios. Recent studies have demonstrated that pre-trained vision-language models like CLIP exhibit strong generalization with just zero or a few normal images. However, existing methods struggle with designing prompt templates, complex token interactions, or requiring additional fine-tuning, resulting in limited flexibility. In this work, we present a simple yet effective method called AdaptCLIP based on two key insights. First, adaptive visual and textual representations should be learned alternately rather than jointly. Second, comparative learning between query and normal image prompt should incorporate both contextual and aligned residual features, rather than relying solely on residual features. AdaptCLIP treats CLIP models as a foundational service, adding only three simple adapters, visual adapter, textual adapter, and prompt-query adapter, at its input or output ends. AdaptCLIP supports zero-/few-shot generalization across domains and possesses a training-free manner on target domains once trained on a base dataset. AdaptCLIP achieves state-of-the-art performance on 12 anomaly detection benchmarks from industrial and medical domains, significantly outperforming existing competitive methods. We will make the code and model of AdaptCLIP available at https://github.com/gaobb/AdaptCLIP.

Figures (15)

• Figure 1: Comparisons of state-of-the-arts and our AdaptCLIP. ✓ means satisfied and ✗ means not satisfied. Our method supports zero-/few-shot (ZS and FS) visual AD across different domains without fine-tuning (FT) on the target dataset. It only adds simple adapters at CLIP's input or output ends without complex token interactions, thus preserving CLIP’s original ability (OA). The AdaptCLIP using only one normal image prompt achieves the best performance in image-level anomaly classification (I-AUROC) and pixel-level anomaly segmentation (P-AUPR) on 12 AD benchmarks from industrial and medical domains. Moreover, the zero-shot AdaptCLIP is also significantly better than existing zero-shot and even some one-shot approaches. The detailed results are reported in Tabs. \ref{['tab:sota_auroc']} and \ref{['tab:sota_aupr']}. Best viewed in zoom.
• Figure 2: The framework of AdaptCLIP, which consists of three pluggable adapters, i.e., visual adapter, textual adapter, and prompt-query adapter. First, the first two adapters alternately learn visual and textual representations for zero-shot anomaly detection (Sec. \ref{['subsec:va']}). The prompt-query adapter further learns a comparison ability between query image and its corresponding normal prompt for few-shot anomaly detection (Sec. \ref{['subsec:pqa']}). Once trained, it can segment any anomalies providing only few-shot and even zero-shot normal image prompts.
• Figure 3: PyTorch pseudocode for the inference of AdaptCLIP.
• Figure 4: Qualitative comparisons of our AdaptCLIP with different prompt numbers on MVTec, VisA, Real-IAD, Kvasir and Endo. More qualitative results of AdaptCLIP can be found in Appendix. Best viewed in color and zoom.
• Figure 5: Image-level anomaly classification and pixel-level anomaly segmentation comparisons of state-of-the-art zero-/one-shot methods and our AdaptCLIP with all three metrics, AUROC, AUPR and F1$_\text{max}$. The one-shot AdaptCLIP utilizes a training-free manner on target domains and achieves more accurate anomaly classification and segmentation on 8 industrial and 4 medical benchmarks.