Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?

TL;DR

The paper addresses the rigidity of traditional industrial anomaly detection (IAD) by leveraging Multimodal Large Language Models (MLLMs) and introducing Echo, a four-expert framework (Reference Extractor, Knowledge Guide, Reasoning Expert, Decision Maker) that integrates retrieval, domain knowledge, and structured reasoning. Echo augments MLLMs with a CLIP-based pipeline, multimodal memory, and context-aware prompts to reduce hallucinations and improve adaptability across diverse industrial scenarios. The approach demonstrates state-of-the-art performance on MMAD benchmarks (MVTec-AD and VisA), delivering notable gains in anomaly discrimination and defect classification while maintaining robustness and interpretability. The work highlights practical impact for real-world industrial deployment by enabling nuanced, context-sensitive anomaly reasoning and decision making through expert-module collaboration and retrieval-augmented guidance.

Abstract

In industrial settings, the accurate detection of anomalies is essential for maintaining product quality and ensuring operational safety. Traditional industrial anomaly detection (IAD) models often struggle with flexibility and adaptability, especially in dynamic production environments where new defect types and operational changes frequently arise. Recent advancements in Multimodal Large Language Models (MLLMs) hold promise for overcoming these limitations by combining visual and textual information processing capabilities. MLLMs excel in general visual understanding due to their training on large, diverse datasets, but they lack domain-specific knowledge, such as industry-specific defect tolerance levels, which limits their effectiveness in IAD tasks. To address these challenges, we propose Echo, a novel multi-expert framework designed to enhance MLLM performance for IAD. Echo integrates four expert modules: Reference Extractor which provides a contextual baseline by retrieving similar normal images, Knowledge Guide which supplies domain-specific insights, Reasoning Expert which enables structured, stepwise reasoning for complex queries, and Decision Maker which synthesizes information from all modules to deliver precise, context-aware responses. Evaluated on the MMAD benchmark, Echo demonstrates significant improvements in adaptability, precision, and robustness, moving closer to meeting the demands of real-world industrial anomaly detection.

Figures (11)

• Figure 1: (a) Directly providing a query image and question to the MLLM may result in hallucinated or wrong outputs. (b) By processing the query image and question through our Echo framework, which integrates reference images, external knowledge, CoT reasoning, and a decision-making module, the system can generate accurate outputs.
• Figure 2: Framework of Echo. The Echo framework uses multiple expert modules to enhance industrial anomaly detection in MLLMs. A query image and question are processed by a CLIP encoder, with the Reference Extractor retrieving a similar normal image and the Knowledge Guide gathering relevant defect information. These inputs are fed into the MLLM, where the Reasoning Expert aids complex inference, and the Decision Maker combines outputs to generate a precise, context-aware response.
• Figure 3: Experts Module. The Experts Module in Echo assigns specific modules to address different types of industrial anomaly detection queries, activating the most relevant experts based on query requirements to deliver precise, context-aware responses.
• Figure 4: Example of 5 subtasks from MMAD, each presented in a multiple-choice format with distractor options designed to test the model's anomaly detection capabilities.
• Figure 5: Multiple Choice, Q$\&$A and True/False evaluation on Anomaly Discrimination.