Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

Jiaqi Zhu, Shaofeng Cai, Fang Deng, Beng Chin Ooi, Junran Wu

TL;DR

ALFA addresses zero-shot visual anomaly detection by leveraging LVLMs for informative prompts and introducing a per-image run-time prompt adaptation strategy with contextual scoring to mitigate cross-semantic ambiguity. It combines this with a training-free fine-grained aligner that projects global image-text alignment into local patch space for precise pixel-level localization, enabling unified global and local VAD. Evaluations on MVTec and VisA show significant improvements over state-of-the-art zero-shot methods in both image-level and pixel-level metrics, and the approach extends to few-shot settings with competitive performance. The work demonstrates the potential of language-guided multimodal models for robust, data-efficient anomaly detection and provides interpretable descriptors via LLM-generated explanations.

Abstract

Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language. Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec and 8.9% on VisA compared to state-of-the-art approaches.

Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

TL;DR

ALFA addresses zero-shot visual anomaly detection by leveraging LVLMs for informative prompts and introducing a per-image run-time prompt adaptation strategy with contextual scoring to mitigate cross-semantic ambiguity. It combines this with a training-free fine-grained aligner that projects global image-text alignment into local patch space for precise pixel-level localization, enabling unified global and local VAD. Evaluations on MVTec and VisA show significant improvements over state-of-the-art zero-shot methods in both image-level and pixel-level metrics, and the approach extends to few-shot settings with competitive performance. The work demonstrates the potential of language-guided multimodal models for robust, data-efficient anomaly detection and provides interpretable descriptors via LLM-generated explanations.

Abstract

Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language. Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec and 8.9% on VisA compared to state-of-the-art approaches.
Paper Structure (18 sections, 8 equations, 6 figures, 7 tables, 1 algorithm)

This paper contains 18 sections, 8 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary
  4. Visual Anomaly Detection
  5. Zero-shot Anomaly Detection with LVLMs
  6. Methodology
  7. Overview
  8. Run-time Prompt Adaptation
  9. Fine-grained Aligner
  10. Experiments
  11. Experimental Setup
  12. Zero-shot anomaly detection
  13. Ablation Study
  14. Interpretability
  15. Few-shot Generalization
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Overview of ALFA, a training-free zero-shot VAD model focusing on vision-language synergy. The first and third prompts are generated by an LLM to describe normal and abnormal images, respectively. The second prompt, however, shows an ambiguous description, posing a challenge in accurately determining the image label, a phenomenon known as cross-semantic ambiguity.
  • Figure 2: Workflow of ALFA with the run-time prompt adaptation strategy, which generates informative prompts and adaptively manages a collection of prompts on a per-image basis via a contextual scoring mechanism. Furthermore, a fine-grained aligner is introduced to generalize the alignment projection from global to local for precise anomaly localization.
  • Figure 3: Overview of the run-time prompt adaptation.
  • Figure 4: Visualization of ALFA's semantic space.
  • Figure 5: Qualitative results of zero-shot VAD. Annotated orange regions indicate detected anomalies, showcasing effective localization of ALFA across diverse anomalies (e.g., broken and bent of varying sizes and quantities) within various classes.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Definition 1: Visual anomaly detection