AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection

TL;DR

This work tackles zero-shot anomaly detection by adapting pre-trained vision-language models with AdaCLIP, a framework that adds static and dynamic prompts (hybrid prompts) to CLIP and employs a hybrid semantic fusion module to improve both pixel- and image-level detection. By training on auxiliary annotated anomaly data and generating per-image dynamic prompts, AdaCLIP exploits cross-domain patterns to detect anomalies in unseen categories. Across 14 industrial and medical datasets, AdaCLIP achieves state-of-the-art results, underscoring the importance of diverse auxiliary data and prompt optimization for generalization. The approach demonstrates effective cross-domain applicability and offers a lightweight, modular path to adapting VLMs for ZSAD without target-category data.

Abstract

Zero-shot anomaly detection (ZSAD) targets the identification of anomalies within images from arbitrary novel categories. This study introduces AdaCLIP for the ZSAD task, leveraging a pre-trained vision-language model (VLM), CLIP. AdaCLIP incorporates learnable prompts into CLIP and optimizes them through training on auxiliary annotated anomaly detection data. Two types of learnable prompts are proposed: static and dynamic. Static prompts are shared across all images, serving to preliminarily adapt CLIP for ZSAD. In contrast, dynamic prompts are generated for each test image, providing CLIP with dynamic adaptation capabilities. The combination of static and dynamic prompts is referred to as hybrid prompts, and yields enhanced ZSAD performance. Extensive experiments conducted across 14 real-world anomaly detection datasets from industrial and medical domains indicate that AdaCLIP outperforms other ZSAD methods and can generalize better to different categories and even domains. Finally, our analysis highlights the importance of diverse auxiliary data and optimized prompts for enhanced generalization capacity. Code is available at https://github.com/caoyunkang/AdaCLIP.

Figures (8)

• Figure 1: Left: Illustrations for training and test data of unsupervised, semi-supervised, and zero-shot anomaly detection paradigms. Right: Quantitative comparison with popular methods by pixel-level max-F1 WinClip on industrial and medical datasets.
• Figure 2: Framework of AdaCLIP.
• Figure 3: Visualization of anomaly maps of different ZSAD methods. The proposed AdaCLIP can get the most precise segmentation results for novel categories in both industrial and medical domains.
• Figure 3: Ablation Results of Static prompts ${\mathbf{P}^S}$ and Dynamic prompts ${\mathbf{P}^D}$.
• Figure 4: Visualization of Patch Embeddings and Anomaly Maps under Different Prompts. PCA is utilized to reduce the dimension of patch embeddings for enhanced visualization. For individual models, the left shows patch embeddings and the right displays anomaly maps.