Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison

Xi Jiang, Yue Guo, Jian Li, Yong Liu, Bin-Bin Gao, Hanqiu Deng, Jun Liu, Heng Zhao, Chengjie Wang, Feng Zheng

Abstract

Multimodal Large Language Models (MLLMs) have achieved impressive success in natural visual understanding, yet they consistently underperform in industrial anomaly detection (IAD). This is because MLLMs trained mostly on general web data differ significantly from industrial images. Moreover, they encode each image independently and can only compare images in the language space, making them insensitive to subtle visual differences that are key to IAD. To tackle these issues, we present AD-Copilot, an interactive MLLM specialized for IAD via visual in-context comparison. We first design a novel data curation pipeline to mine inspection knowledge from sparsely labeled industrial images and generate precise samples for captioning, VQA, and defect localization, yielding a large-scale multimodal dataset Chat-AD rich in semantic signals for IAD. On this foundation, AD-Copilot incorporates a novel Comparison Encoder that employs cross-attention between paired image features to enhance multi-image fine-grained perception, and is trained with a multi-stage strategy that incorporates domain knowledge and gradually enhances IAD skills. In addition, we introduce MMAD-BBox, an extended benchmark for anomaly localization with bounding-box-based evaluation. The experiments show that AD-Copilot achieves 82.3% accuracy on the MMAD benchmark, outperforming all other models without any data leakage. In the MMAD-BBox test, it achieves a maximum improvement of $3.35\times$ over the baseline. AD-Copilot also exhibits excellent generalization of its performance gains across other specialized and general-purpose benchmarks. Remarkably, AD-Copilot surpasses human expert-level performance on several IAD tasks, demonstrating its potential as a reliable assistant for real-world industrial inspection. All datasets and models will be released for the broader benefit of the community.

AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison

Abstract

Multimodal Large Language Models (MLLMs) have achieved impressive success in natural visual understanding, yet they consistently underperform in industrial anomaly detection (IAD). This is because MLLMs trained mostly on general web data differ significantly from industrial images. Moreover, they encode each image independently and can only compare images in the language space, making them insensitive to subtle visual differences that are key to IAD. To tackle these issues, we present AD-Copilot, an interactive MLLM specialized for IAD via visual in-context comparison. We first design a novel data curation pipeline to mine inspection knowledge from sparsely labeled industrial images and generate precise samples for captioning, VQA, and defect localization, yielding a large-scale multimodal dataset Chat-AD rich in semantic signals for IAD. On this foundation, AD-Copilot incorporates a novel Comparison Encoder that employs cross-attention between paired image features to enhance multi-image fine-grained perception, and is trained with a multi-stage strategy that incorporates domain knowledge and gradually enhances IAD skills. In addition, we introduce MMAD-BBox, an extended benchmark for anomaly localization with bounding-box-based evaluation. The experiments show that AD-Copilot achieves 82.3% accuracy on the MMAD benchmark, outperforming all other models without any data leakage. In the MMAD-BBox test, it achieves a maximum improvement of over the baseline. AD-Copilot also exhibits excellent generalization of its performance gains across other specialized and general-purpose benchmarks. Remarkably, AD-Copilot surpasses human expert-level performance on several IAD tasks, demonstrating its potential as a reliable assistant for real-world industrial inspection. All datasets and models will be released for the broader benefit of the community.
Paper Structure (37 sections, 1 equation, 7 figures, 11 tables)

This paper contains 37 sections, 1 equation, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Industrial Anomaly Detection with Foundation Models
  4. Visual Comparison and Change Detection
  5. Chat-AD Dataset
  6. Query-Template Pairing: The Foundation of Visual In-context Comparison
  7. Contrastive Description Generation
  8. Multi-task Dialogue and Verifiable QA Data
  9. Model Architecture
  10. Motivation: The Semantic Comparison Bottleneck
  11. The Comparison Encoder
  12. Architecture Design
  13. Design Rationale
  14. Structure Details
  15. Model Training
  16. ...and 22 more sections

Figures (7)

  • Figure 1: AD-Copilot was trained on meticulously curated datasets, including high-quality synthetic IAD caption and VQA data, together with general-domain data aligned with IAD objectives. Experiment results show that AD-Copilot delivers strong performance across a range of IAD tasks. Case studies on downstream applications further highlight its practical utility and substantial potential in real-world manufacturing settings.
  • Figure 2: The pipeline for Chat-AD synthesis. Each query image is paired with a normal template image, and the generated text explicitly describes visual differences between them, establishing comparison as the foundational data paradigm. $^{\ast}$ Currently private, upcoming public dataset.
  • Figure 3: The model architecture of AD-Copilot. The Comparison Encoder performs cross-attention between paired image features to generate comparison tokens, enabling visual in-context comparison at the encoding stage.
  • Figure 4: Task and metric of MMAD-BBox.
  • Figure 5: Data scaling behavior of AD-Copilot. Performance improves with more data, but removing the Comparison Encoder or using lower-quality annotations causes large drops.
  • ...and 2 more figures