ADSeeker: A Knowledge-Grounded Reasoning Framework for Industry Anomaly Detection and Reasoning

Abstract

Automatic vision inspection holds significant importance in industry inspection. While multimodal large language models (MLLMs) exhibit strong language understanding capabilities and hold promise for this task, their performance remains significantly inferior to that of human experts. In this context, we identify two key challenges: (i) insufficient integration of anomaly detection (AD) knowledge during pre-training, and (ii) the lack of technically precise and context-aware language generation for anomaly reasoning. To address these issues, we propose ADSeeker, an anomaly task assistant designed to enhance inspection performance through knowledge-grounded reasoning. ADSeeker first leverages a curated visual document knowledge base, SEEK-M&V, which we construct to address the limitations of existing resources that rely solely on unstructured text. SEEK-M\&V includes semantic-rich descriptions and image-document pairs, enabling more comprehensive anomaly understanding. To effectively retrieve and utilize this knowledge, we introduce the Query Image-Knowledge Retrieval-Augmented Generation Q2K RAG framework. To further enhance the performance in zero-shot anomaly detection (ZSAD), ADSeeker leverages the Hierarchical Sparse Prompt mechanism and type-level features to efficiently extract anomaly patterns. Furthermore, to tackle the challenge of limited industry anomaly detection (IAD) data, we introduce the largest-scale AD dataset, Multi-type Anomaly MulA, encompassing 72 multi-scale defect types across 26 categories. Extensive experiments show that our plug-and-play framework, ADSeeker, achieves state-of-the-art zero-shot performance on several benchmark datasets.

Figures (6)

• Figure 1: Visualization of the reasoning capability. Comparative results of ADSeeker against state-of-the-art models on the MMAD benchmark. Notably, the framework is specifically designed for IAD, its performance on defect analysis and localization tasks is better than any other IAD models.
• Figure 2: Industrial and medical image anomaly reasoning results of ADSeeker. The results include accurate anomaly location, anomaly description, defect classification, and other fine-grained information.
• Figure 3: Details of MulA Dataset. (A) Statistics and components of the MulA Dataset, the newest dataset with the largest number and most types, and (B) examples of different types of type-level feature. Our dataset can also be organized by its defect type to enhance the Anomaly Detection performance of our ADSeeker through clustering.
• Figure 4: The architecture of ADSeeker. It consists of two main knowledge-infused pathways: (1) The query image and knowledge documents are fed to the Q2K RAG to retrieve high-relevant domain knowledge. (2) The AD Expert integrates defect-region information and type-level features into semantic-rich visual tokens which will be passed into VLM.
• Figure 5: Illustration of the core architecture of Q2K RAG. Solving the problem of repetitive retrieval in industrial knowledge base through KDE-Sample strategy.