PromptMAD: Cross-Modal Prompting for Multi-Class Visual Anomaly Localization

TL;DR

PromptMAD tackles unsupervised multi-class visual anomaly detection by integrating semantic cross-modal prompts via CLIP with a text-guided diffusion-based reconstruction and a deep text-guided segmentor. The approach combines a focal loss to address pixel-level class imbalance and a diffusion refinement module to produce high-resolution anomaly maps, enabling precise localization of camouflaged defects across diverse categories. Evaluations on MVTec-AD demonstrate state-of-the-art pixel-level localization with robust cross-class generalization, while maintaining practical inference speed suitable for industrial settings. This work advances unified anomaly detection by leveraging semantic guidance alongside reconstruction, achieving stronger boundary precision and reduced false positives without increasing inference cost significantly.

Abstract

Visual anomaly detection in multi-class settings poses significant challenges due to the diversity of object categories, the scarcity of anomalous examples, and the presence of camouflaged defects. In this paper, we propose PromptMAD, a cross-modal prompting framework for unsupervised visual anomaly detection and localization that integrates semantic guidance through vision-language alignment. By leveraging CLIP-encoded text prompts describing both normal and anomalous class-specific characteristics, our method enriches visual reconstruction with semantic context, improving the detection of subtle and textural anomalies. To further address the challenge of class imbalance at the pixel level, we incorporate Focal loss function, which emphasizes hard-to-detect anomalous regions during training. Our architecture also includes a supervised segmentor that fuses multi-scale convolutional features with Transformer-based spatial attention and diffusion iterative refinement, yielding precise and high-resolution anomaly maps. Extensive experiments on the MVTec-AD dataset demonstrate that our method achieves state-of-the-art pixel-level performance, improving mean AUC to 98.35% and AP to 66.54%, while maintaining efficiency across diverse categories.

Figures (2)

• Figure 1: Overview of PromptMAD architecture for multi-class visual anomaly localization. A pre-trained backbone extracts multi-scale features from target and normal prompt images. Cross-modal prompts fuse visual tokens with CLIP-encoded textual descriptions of normal and anomalous class traits (e.g., "uniform threads, no deformations" vs. "bent leads, cracks") for semantic guidance. The bidirectional Transformer decoder reconstructs normal features while amplifying anomalies. Reconstruction errors feed into an enhanced supervised refiner with deeper CNNs for multi-scale extraction, Transformer attention for spatial focus, and diffusion-based iterative refinement for high-resolution anomaly maps.
• Figure 2: Qualitative comparison of anomaly localization on selected MVTec-AD test examples (cable, hazelnut, metal nut, pill, tile, toothbrush, transistor, wood, zipper). From top to bottom: input anomalous images, ground truth masks, OneNIP onenip anomaly maps, our PromptMAD anomaly maps.