IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection

TL;DR

To enhance the visual grounding ability of MLLMs, this article proposes text-guided enhancer (TGE), wherein image features interact with normal and abnormal text prompts to dynamically select enhancement pathways, which enables language models to focus on the specific aspects of visual data, enhancing their ability to accurately interpret and respond to anomalies within images.

Abstract

The robust causal capability of Multimodal Large Language Models (MLLMs) hold the potential of detecting defective objects in Industrial Anomaly Detection (IAD). However, most traditional IAD methods lack the ability to provide multi-turn human-machine dialogues and detailed descriptions, such as the color of objects, the shape of an anomaly, or specific types of anomalies. At the same time, methods based on large pre-trained models have not fully stimulated the ability of large models in anomaly detection tasks. In this paper, we explore the combination of rich text semantics with both image-level and pixel-level information from images and propose IAD-GPT, a novel paradigm based on MLLMs for IAD. We employ Abnormal Prompt Generator (APG) to generate detailed anomaly prompts for specific objects. These specific prompts from the large language model (LLM) are used to activate the detection and segmentation functions of the pre-trained visual-language model (i.e., CLIP). To enhance the visual grounding ability of MLLMs, we propose Text-Guided Enhancer, wherein image features interact with normal and abnormal text prompts to dynamically select enhancement pathways, which enables language models to focus on specific aspects of visual data, enhancing their ability to accurately interpret and respond to anomalies within images. Moreover, we design a Multi-Mask Fusion module to incorporate mask as expert knowledge, which enhances the LLM's perception of pixel-level anomalies. Extensive experiments on MVTec-AD and VisA datasets demonstrate our state-of-the-art performance on self-supervised and few-shot anomaly detection and segmentation tasks, such as MVTec-AD and VisA datasets. The codes are available at \href{https://github.com/LiZeWen1225/IAD-GPT}{https://github.com/LiZeWen1225/IAD-GPT}.

Figures (8)

• Figure 1: Comparison between our IAD-GPT, traditional IAD methods and AnomalyGPT. (a) Traditional methods use separate models for different classes and provide anomaly scores only. (b) Unified methods manage to accomplish anomaly detection for various classes with a unified framework. (c) AnomalyGPT, based on the settings in (b), enhances the pixel-level visual knowledge of MLLMs to perceive anomalies. (d) IAD-GPT provides GPT-generated abnormal text to improve localization capabilities and enhances image-level and pixel-level visual knowledge to achieve better anomaly recognition by MLLMs.
• Figure 2: Overview of IAD-GPT. Abnormal Prompt Generator provides category specific text prompts for decoder. Text-Guided Enhancer and Multi-Mask Fusion provide image-level visual information and pixel-level expert knowledge to LLMs, respectively.
• Figure 3: Example of APG for leather. We improve the stability of LLM-generated prompts by designing an QA session and providing illustrative examples.
• Figure 4: Visualization comparison of anomaly image generation results between NSA and CutPaste methods. Red box indicates abnormal area.
• Figure 5: Illustration of generating abnormal prompts and a 3x3 grid of images for LLM to answer abnormal locations. We first input the answer template into LLM to generate diversified answers to improve the diversity and stability of model training. Then randomly select a template and fill the location information of the generated abnormal image into the answer.