CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection

TL;DR

This work tackles anomaly detection under zero-shot and few-shot constraints by fusing discriminative CLIP signals with diffusion-based local features. The proposed CLIPFUSION framework comprises PatchCLIP for global cues and MapDiff for local details, combining language-guided heatmaps and diffusion cross-attention maps into a unified score map via $M := \alpha (M^{\texttt{CLIP}, L}+M^{\texttt{CLIP}, V}) + (1-\alpha) (M^{\texttt{Diff}, L}+M^{\texttt{Diff}, V})$. In zero-shot, only language-guided maps are used, while few-shot benefits come from memory-bank comparisons of features from both models. Experiments on MVTec-AD and VisA show state-of-the-art performance in anomaly segmentation and classification across zero-/few-shot settings, with faster inference than competitive baselines. The results highlight the practical value of multi-modal, multi-model fusion for scalable, real-world anomaly detection and suggest broader applicability to other recognition tasks.

Abstract

Anomaly detection is a complex problem due to the ambiguity in defining anomalies, the diversity of anomaly types (e.g., local and global defect), and the scarcity of training data. As such, it necessitates a comprehensive model capable of capturing both low-level and high-level features, even with limited data. To address this, we propose CLIPFUSION, a method that leverages both discriminative and generative foundation models. Specifically, the CLIP-based discriminative model excels at capturing global features, while the diffusion-based generative model effectively captures local details, creating a synergistic and complementary approach. Notably, we introduce a methodology for utilizing cross-attention maps and feature maps extracted from diffusion models specifically for anomaly detection. Experimental results on benchmark datasets (MVTec-AD, VisA) demonstrate that CLIPFUSION consistently outperforms baseline methods, achieving outstanding performance in both anomaly segmentation and classification. We believe that our method underscores the effectiveness of multi-modal and multi-model fusion in tackling the multifaceted challenges of anomaly detection, providing a scalable solution for real-world applications.

Figures (8)

• Figure 1: Qualitative results of our model CLIPFUSION. It captures fine-grained anomalies more effectively than WinCLIP.
• Figure 2: Overall framework for CLIPFUSION. The CLIP-based model and the diffusion-based model process query and reference images to generate anomaly maps. The final anomaly map is obtained by fusing the outputs of the vision and language components of both models.
• Figure 3: Examples of cross-attention maps extracted from a diffusion denoiser.
• Figure 4: AUROC across training epochs on the VisA dataset (1-shot setting).
• Figure 5: Failure analysis examples.