Analyzing VLM-Based Approaches for Anomaly Classification and Segmentation

TL;DR

This study benchmarked two vision-language model (VLM) approaches for industrial anomaly detection: WinCLIP, which uses sliding-window dense feature extraction with a compositional prompt ensemble, and AnomalyCLIP, which employs object-agnostic prompt learning with multi-level text refinement and DPAM to enhance localization. AnomalyCLIP achieved substantially higher average performance across image-level classification and pixel-level segmentation on the MVTec AD benchmark, with a Classification AUROC of $0.916$ and Segmentation AUROC of $0.907$, outperforming WinCLIP’s $0.612$ and $0.726$, respectively. The results highlight that learned prompts and attention modulation improve zero-shot generalization and defect localization, especially for texture-rich categories, while window-based methods retain value for fine-grained localization in certain defect types. The work suggests a promising direction in combining learned textual representations with explicit spatial reasoning to build scalable, robust industrial inspection systems, and discusses future hybrids, multi-scale fusion, and domain adaptation to broaden applicability.

Abstract

Vision-Language Models (VLMs), particularly CLIP, have revolutionized anomaly detection by enabling zero-shot and few-shot defect identification without extensive labeled datasets. By learning aligned representations of images and text, VLMs facilitate anomaly classification and segmentation through natural language descriptions of normal and abnormal states, eliminating traditional requirements for task-specific training or defect examples. This project presents a comprehensive analysis of VLM-based approaches for anomaly classification (AC) and anomaly segmentation (AS). We systematically investigate key architectural paradigms including sliding window-based dense feature extraction (WinCLIP), multi-stage feature alignment with learnable projections (AprilLab framework), and compositional prompt ensemble strategies. Our analysis evaluates these methods across critical dimensions: feature extraction mechanisms, text-visual alignment strategies, prompt engineering techniques, zero-shot versus few-shot trade-offs, computational efficiency, and cross-domain generalization. Through rigorous experimentation on benchmarks such as MVTec AD and VisA, we compare classification accuracy, segmentation precision, and inference efficiency. The primary contribution is a foundational understanding of how and why VLMs succeed in anomaly detection, synthesizing practical insights for method selection and identifying current limitations. This work aims to facilitate informed adoption of VLM-based methods in industrial quality control and guide future research directions.

Figures (5)

• Figure 1: Classification performance (AUROC) comparison between AnomalyCLIP and WinCLIP across all 15 MVTec AD categories. Dashed lines indicate average performance (AnomalyCLIP: 0.916, WinCLIP: 0.612).
• Figure 2: Segmentation performance (AUROC) comparison between AnomalyCLIP and WinCLIP across all 15 MVTec AD categories. Dashed lines indicate average performance (AnomalyCLIP: 0.907, WinCLIP: 0.726).
• Figure 3: Direct comparison of WinCLIP zero-shot vs few-shot (4-shot) performance on image-level anomaly detection (AUROC) across MVTec AD categories.
• Figure 4: Direct comparison of WinCLIP zero-shot vs few-shot (4-shot) performance on pixel-level anomaly detection (AUROC) across MVTec AD categories.
• Figure 5: Impact of window size on WinCLIP's few-shot image-level anomaly detection performance (AUROC) across all MVTec AD categories. Window sizes range from 1 (single pixel) to 7.