Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection

Xuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yong Liu

TL;DR

A framework called CLIP-AD is proposed to leverage the zero-shot capabilities of the large vision-language model CLIP to leverage the zero-shot capabilities of the large vision-language model CLIP, and a Staged Dual-Path model (SDP) is introduced that leverages features from various levels and applies architecture and feature surgery.

Abstract

This paper considers zero-shot Anomaly Detection (AD), performing AD without reference images of the test objects. We propose a framework called CLIP-AD to leverage the zero-shot capabilities of the large vision-language model CLIP. Firstly, we reinterpret the text prompts design from a distributional perspective and propose a Representative Vector Selection (RVS) paradigm to obtain improved text features. Secondly, we note opposite predictions and irrelevant highlights in the direct computation of the anomaly maps. To address these issues, we introduce a Staged Dual-Path model (SDP) that leverages features from various levels and applies architecture and feature surgery. Lastly, delving deeply into the two phenomena, we point out that the image and text features are not aligned in the joint embedding space. Thus, we introduce a fine-tuning strategy by adding linear layers and construct an extended model SDP+, further enhancing the performance. Abundant experiments demonstrate the effectiveness of our approach, e.g., on MVTec-AD, SDP outperforms the SOTA WinCLIP by +4.2/+10.7 in segmentation metrics F1-max/PRO, while SDP+ achieves +8.3/+20.5 improvements.

CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection

TL;DR

A framework called CLIP-AD is proposed to leverage the zero-shot capabilities of the large vision-language model CLIP to leverage the zero-shot capabilities of the large vision-language model CLIP, and a Staged Dual-Path model (SDP) is introduced that leverages features from various levels and applies architecture and feature surgery.

Abstract

This paper considers zero-shot Anomaly Detection (AD), performing AD without reference images of the test objects. We propose a framework called CLIP-AD to leverage the zero-shot capabilities of the large vision-language model CLIP. Firstly, we reinterpret the text prompts design from a distributional perspective and propose a Representative Vector Selection (RVS) paradigm to obtain improved text features. Secondly, we note opposite predictions and irrelevant highlights in the direct computation of the anomaly maps. To address these issues, we introduce a Staged Dual-Path model (SDP) that leverages features from various levels and applies architecture and feature surgery. Lastly, delving deeply into the two phenomena, we point out that the image and text features are not aligned in the joint embedding space. Thus, we introduce a fine-tuning strategy by adding linear layers and construct an extended model SDP+, further enhancing the performance. Abundant experiments demonstrate the effectiveness of our approach, e.g., on MVTec-AD, SDP outperforms the SOTA WinCLIP by +4.2/+10.7 in segmentation metrics F1-max/PRO, while SDP+ achieves +8.3/+20.5 improvements.
Paper Structure (18 sections, 16 equations, 9 figures, 12 tables)

This paper contains 18 sections, 16 equations, 9 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Anomaly Detection
  4. Vision-Language Models
  5. Methodology of CLIP-AD
  6. Text prompts design based on RVS
  7. Zero-Shot AD without Fine-tuning
  8. Zero-Shot AD with Fine-tuning
  9. Experiments
  10. Experimental Setups
  11. Comparison with State-of-the-Arts
  12. Ablation Study
  13. Conclusion
  14. More Details about RVS
  15. Reasons for the Misalignment
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Visualization of the two unexpected phenomena, opposite predictions and irrelevant highlights, generated by directly computing (Comp. Directly) the anomaly maps.
  • Figure 2: Mapping the entire image feature maps to the joint embedding space using a linear layer (linear mapping).
  • Figure 3: Overview of our CLIP-AD framework that contains: 1) the blue arrows in the lower section represent the processing steps of SDP; 2) the red arrows in the upper section depict the processing steps of SDP+. For the same category, the text prompts are consistent. $\oplus$ and $\otimes$ represent pixel-level addition and multiplication, respectively.
  • Figure 4: Structure of the dual-path block. "ViT Layer" represents the original layers in ViT, while "Surgery Layer" refers to the new layers altered through architecture surgery.
  • Figure 5: Qualitative comparisons on the two industrial and four medical datasets, with MVTec-AD offering five examples and all others providing two each. The order from left to right is MVTec-AD, VisA, ISIC, CVC-ClinicDB, HeadCT, and BrainMRI.
  • ...and 4 more figures