Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models

Sassan Mokhtar, Arian Mousakhan, Silvio Galesso, Jawad Tayyub, Thomas Brox

TL;DR

This work tackles the need for semantic classification of industrial anomalies beyond mere detection by introducing VELM, a pipeline that couples a fast vision-based anomaly detector with a multimodal LLM classifier. To enable rigorous evaluation, it refines two standard benchmarks into MVTec-AC and VisA-AC with precise anomaly-class labels. Across MVTec-AD, MVTec-AC, and VisA-AC, VELM demonstrates strong anomaly-classification performance and highlights the critical role of accurate localization and carefully designed multimodal prompts. The approach offers a practical, flexible framework for real-world inspection by distinguishing defects from benign deviations and by enabling user-defined classification criteria without task-specific training.

Abstract

Recent advances in visual industrial anomaly detection have demonstrated exceptional performance in identifying and segmenting anomalous regions while maintaining fast inference speeds. However, anomaly classification-distinguishing different types of anomalies-remains largely unexplored despite its critical importance in real-world inspection tasks. To address this gap, we propose VELM, a novel LLM-based pipeline for anomaly classification. Given the critical importance of inference speed, we first apply an unsupervised anomaly detection method as a vision expert to assess the normality of an observation. If an anomaly is detected, the LLM then classifies its type. A key challenge in developing and evaluating anomaly classification models is the lack of precise annotations of anomaly classes in existing datasets. To address this limitation, we introduce MVTec-AC and VisA-AC, refined versions of the widely used MVTec-AD and VisA datasets, which include accurate anomaly class labels for rigorous evaluation. Our approach achieves a state-of-the-art anomaly classification accuracy of 80.4% on MVTec-AD, exceeding the prior baselines by 5%, and 84% on MVTec-AC, demonstrating the effectiveness of VELM in understanding and categorizing anomalies. We hope our methodology and benchmark inspire further research in anomaly classification, helping bridge the gap between detection and comprehensive anomaly characterization.

Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models

TL;DR

This work tackles the need for semantic classification of industrial anomalies beyond mere detection by introducing VELM, a pipeline that couples a fast vision-based anomaly detector with a multimodal LLM classifier. To enable rigorous evaluation, it refines two standard benchmarks into MVTec-AC and VisA-AC with precise anomaly-class labels. Across MVTec-AD, MVTec-AC, and VisA-AC, VELM demonstrates strong anomaly-classification performance and highlights the critical role of accurate localization and carefully designed multimodal prompts. The approach offers a practical, flexible framework for real-world inspection by distinguishing defects from benign deviations and by enabling user-defined classification criteria without task-specific training.

Abstract

Recent advances in visual industrial anomaly detection have demonstrated exceptional performance in identifying and segmenting anomalous regions while maintaining fast inference speeds. However, anomaly classification-distinguishing different types of anomalies-remains largely unexplored despite its critical importance in real-world inspection tasks. To address this gap, we propose VELM, a novel LLM-based pipeline for anomaly classification. Given the critical importance of inference speed, we first apply an unsupervised anomaly detection method as a vision expert to assess the normality of an observation. If an anomaly is detected, the LLM then classifies its type. A key challenge in developing and evaluating anomaly classification models is the lack of precise annotations of anomaly classes in existing datasets. To address this limitation, we introduce MVTec-AC and VisA-AC, refined versions of the widely used MVTec-AD and VisA datasets, which include accurate anomaly class labels for rigorous evaluation. Our approach achieves a state-of-the-art anomaly classification accuracy of 80.4% on MVTec-AD, exceeding the prior baselines by 5%, and 84% on MVTec-AC, demonstrating the effectiveness of VELM in understanding and categorizing anomalies. We hope our methodology and benchmark inspire further research in anomaly classification, helping bridge the gap between detection and comprehensive anomaly characterization.
Paper Structure (23 sections, 3 equations, 6 figures, 5 tables)

This paper contains 23 sections, 3 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Anomaly Detection
  4. Semantic Characterization of Anomalies
  5. Method
  6. Vision Expert
  7. Multimodal LLM-based Anomaly Classifier
  8. Dataset Refinement for Anomaly Classification
  9. MVTec-AC
  10. VisA-AC
  11. Experiments
  12. Evaluation Metrics
  13. Macro Accuracy
  14. Macro F1-score
  15. Cohen’s Kappa
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Example of the proposed use case for our anomaly detection and classification pipeline. In most cases (1), the test samples will be directly deemed healthy by a visual detector (e.g. PatchCore, DDAD, etc.). If not, a semantic enabled multi-modal model will decide whether the defect is admissible (2) or not (3), based on user-defined instructions.
  • Figure 2: The figure illustrates four anomaly cases detected by a visual anomaly detector mousakhan2023anomaly. On the left, clear defects include bent wires and missing cables. On the right, anomalies may not indicate actual faults, such as a color change due to design updates or a minor indentation. Our LLM-based model helps distinguish critical issues from benign variations, ensuring informed decision-making after detection.
  • Figure 3: Overview of VELM. Given a query image, VELM first processes it using a Vision Expert, which performs both anomaly detection and localization. If the image is classified as normal, the process terminates. Otherwise, based on the localization from the Vision Expert, a visual prompt is generated by overlaying a red contour on the detected anomaly. The Multimodal-LLM then receives the normal image, query image, visual prompt, and a textual prompt to classify the anomaly into predefined categories
  • Figure 4: Example of a structured text prompt used for anomaly classification with multimodal LLMs. The prompt includes a normal object description, anomaly class definitions, and a classification strategy to guide the model's decision-making
  • Figure 5: Examples of misclassified samples in the MVTec-AD dataset. The first column displays bent samples, and the second column shows broken samples. However, the third column contains broken samples incorrectly labeled as bent, highlighting the need for dataset refinement.
  • ...and 1 more figures