MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection

TL;DR

MMAD addresses the need for a domain-specific benchmark to evaluate multimodal large language models in industrial anomaly detection by introducing a comprehensive dataset and VQA-style evaluation across seven subtasks. It combines four public IAD datasets to yield 8,366 images and 244 defect types, and generates 39,672 questions via a GPT-4V-driven pipeline with manual filtering. The study benchmarks a wide range of SOTA MLLMs, finding GPT-4o to lead at 74.9% average accuracy but still fall short for industrial requirements, particularly on anomaly-related tasks; it also tests training-free boosts like retrieval-augmented generation (RAG) and an Expert Agent, observing mixed but insightful gains. The work demonstrates MMAD’s value as a rigorous testbed, highlights the need for more industrial data and domain knowledge, and points to practical research directions for improving cross-image reasoning and domain-specific capabilities in MLLMs for real-world IAD.

Abstract

In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research.

Figures (17)

• Figure 1: Left: Innermost layer: image components, middle layer: subtasks composition, outermost layer: object categories. MMAD covers 7 key subtasks and 38 representative categories of IAD. Right: Results of 5 representative MLLMs and Human. The left-skewness indicates that models perform well on object-related questions but poorly on questions related to defects.
• Figure 2: Examples of 7 subtasks of MMAD. Each question is presented in a multiple-choice format and includes several distractor options. We present different categories of objects in various examples to demonstrate the diversity.
• Figure 3: The VQA data generation pipeline for IAD. We utilize images from the open-source IAD dataset and leverage GPT-4V to automate the generation of question-answer texts. Initially, the model is prompted to provide detailed captions for IAD images by summarizing visual cues and textual prior knowledge. Based on these image-caption pairs, the model then generates questions across different subtasks according to predefined question definitions and examples, simultaneously creating multiple-choice questions with several distractor options. Finally, manual verification is conducted to filter out low-quality VQA pairs, resulting in high-quality VQA data for the IAD.
• Figure 4: Illustration of two proposed boost methods of MLLMs in MMAD.
• Figure 5: Left: Scaling law of model size in MMAD evaluation. We use the InternVL2 series as examples. Right: The performance trends of InternVL2-76B with varying counts of normal samples.