Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anomaly Detection by Adapting a pre-trained Vision Language Model

Yuxuan Cai, Xinwei He, Dingkang Liang, Ao Tong, Xiang Bai

TL;DR

A unified framework named CLIP-ADA for Anomaly Detection by Adapting a pre-trained CLIP model and introduces the learnable prompt and proposes to associate it with abnormal patterns through self-supervised learning to fully exploit the representation power of CLIP.

Abstract

Recently, large vision and language models have shown their success when adapting them to many downstream tasks. In this paper, we present a unified framework named CLIP-ADA for Anomaly Detection by Adapting a pre-trained CLIP model. To this end, we make two important improvements: 1) To acquire unified anomaly detection across industrial images of multiple categories, we introduce the learnable prompt and propose to associate it with abnormal patterns through self-supervised learning. 2) To fully exploit the representation power of CLIP, we introduce an anomaly region refinement strategy to refine the localization quality. During testing, the anomalies are localized by directly calculating the similarity between the representation of the learnable prompt and the image. Comprehensive experiments demonstrate the superiority of our framework, e.g., we achieve the state-of-the-art 97.5/55.6 and 89.3/33.1 on MVTec-AD and VisA for anomaly detection and localization. In addition, the proposed method also achieves encouraging performance with marginal training data, which is more challenging.

Anomaly Detection by Adapting a pre-trained Vision Language Model

TL;DR

A unified framework named CLIP-ADA for Anomaly Detection by Adapting a pre-trained CLIP model and introduces the learnable prompt and proposes to associate it with abnormal patterns through self-supervised learning to fully exploit the representation power of CLIP.

Abstract

Recently, large vision and language models have shown their success when adapting them to many downstream tasks. In this paper, we present a unified framework named CLIP-ADA for Anomaly Detection by Adapting a pre-trained CLIP model. To this end, we make two important improvements: 1) To acquire unified anomaly detection across industrial images of multiple categories, we introduce the learnable prompt and propose to associate it with abnormal patterns through self-supervised learning. 2) To fully exploit the representation power of CLIP, we introduce an anomaly region refinement strategy to refine the localization quality. During testing, the anomalies are localized by directly calculating the similarity between the representation of the learnable prompt and the image. Comprehensive experiments demonstrate the superiority of our framework, e.g., we achieve the state-of-the-art 97.5/55.6 and 89.3/33.1 on MVTec-AD and VisA for anomaly detection and localization. In addition, the proposed method also achieves encouraging performance with marginal training data, which is more challenging.
Paper Structure (33 sections, 6 equations, 6 figures, 6 tables)

This paper contains 33 sections, 6 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Realted Works
  3. Anomaly Detection
  4. Unified Anomaly Detection
  5. Vision-Language Contrastive Learning
  6. Preliminary
  7. A revisit of CLIP
  8. Problem Definition
  9. Method
  10. Frozen CLIP as the Backbone
  11. Anomaly Adaptation with Learnable Prompts
  12. Optimizing Objective.
  13. Anomaly Region Refinement
  14. Overall Objective and Inference
  15. Overall Objective.
  16. ...and 18 more sections

Figures (6)

  • Figure 1: (a) Existing CLIP-Based anomaly detection methods, which directly utilize hand-craft text templates and leverage the similarity between the image and text features to spot anomalies; (b) The proposed CLIP-ADA enables more accurate anomaly detection and localization results by incorporating learnable text prompts and a coarse-to-fine strategy.
  • Figure 2: An overview of our method. We use the frozen CLIP as our backbone. In the image branch, we extract visual features through the CLIP image encoder, followed by a linear layer for adaptation. In the text branch, we combine the pre-defined and learnable prompts, and then employ a CLIP text encoder to derive the text embedding. Subsequently, we calculate the similarity map between the text embedding and visual features, which acts as the initial localization result. To improve the quality of the initial prediction, we further propose a refinement strategy.
  • Figure 3: Visualization of localization results on MVTec.
  • Figure 4: Visualization of localization results on VisA.
  • Figure 5: Visualization of localization results with different refinement layers.
  • ...and 1 more figures