Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models

TL;DR

This work tackles zero-shot anomaly detection with multimodal large language models by first establishing Anomaly-Instruct-125k and VisA-D&R to support instruction tuning and evaluation for visual anomaly reasoning. It introduces Anomaly-OneVision, a specialist visual assistant built on a base MLLM (LLaVA-OneVision) and guided by a dedicated anomaly expert, employing Look-Twice Feature Matching and a Visual Token Selector to focus on suspicious visual tokens. The two-stage training, plus a large-scale WebAD data collection, enables robust zero-shot detection and detailed anomaly reasoning across industrial, 3D, and medical domains, with significant improvements over existing generalist MLLMs. The work provides a practical path toward interpretable, reliable visual inspection systems and lays groundwork for unified AD and reasoning in diverse domains, with extensions to 3D and medical data outlined for future study.

Abstract

Zero-Shot Anomaly Detection (ZSAD) is an emerging AD paradigm. Unlike the traditional unsupervised AD setting that requires a large number of normal samples to train a model, ZSAD is more practical for handling data-restricted real-world scenarios. Recently, Multimodal Large Language Models (MLLMs) have shown revolutionary reasoning capabilities in various vision tasks. However, the reasoning of image abnormalities remains underexplored due to the lack of corresponding datasets and benchmarks. To facilitate research in AD & reasoning, we establish the first visual instruction tuning dataset, Anomaly-Instruct-125k, and the evaluation benchmark, VisA-D&R. Through investigation with our benchmark, we reveal that current MLLMs like GPT-4o cannot accurately detect and describe fine-grained anomalous details in images. To address this, we propose Anomaly-OneVision (Anomaly-OV), the first specialist visual assistant for ZSAD and reasoning. Inspired by human behavior in visual inspection, Anomaly-OV leverages a Look-Twice Feature Matching (LTFM) mechanism to adaptively select and emphasize abnormal visual tokens. Extensive experiments demonstrate that Anomaly-OV achieves significant improvements over advanced generalist models in both detection and reasoning. Extensions to medical and 3D AD are provided for future study. The link to our project page: https://xujiacong.github.io/Anomaly-OV/

Figures (10)

• Figure 1: Visualization of the image-level AUROC comparison between our Anomaly-OV and current state-of-the-art ZSAD methods (WinCLIP jeong2023winclip, AnoVL deng2023anovl, AnomalyCLIP zhou2024anomalyclip, AdaCLIP cao2025adaclip). Notably, our zero-shot performance on VisA even surpasses most recent advances in the few-shot setting li2024promptadzhu2024towardgu2024anomalygpt.
• Figure 2: Industrial image anomaly reasoning results from GPT-4o gpt-api-4o and our Anomaly-OV. The responses for fine-grained anomaly reasoning are highlighted, with the ground truth given for reference.
• Figure 3: Overview of the Anomaly-OV architecture. It consists of two training stages: (1) professional training for the anomaly expert, and (2) visual instruction tuning for anomaly detection and reasoning. Text and visual tokens are distinguished by different colors.
• Figure 4: Simulation of visual anomaly inspection by humans.
• Figure 5: Composition of the instruction data in Anomaly-Instruct-125k. There are four main types of image samples: in-the-wild, industrial, medical, and 3D (in the format of multi-view images), covering most image anomaly detection tasks and enabling the possibility of a unified assistant for visual inspection. The reasoning words are highlighted in blue. For more information about dataset establishment, statistics, and the data collection pipeline, please refer to Section \ref{['sup_dataset']} in the supplementary.