Referring Industrial Anomaly Segmentation

TL;DR

This work presents Referring Industrial Anomaly Segmentation (RIAS), a language-guided approach to detect and segment industrial anomalies with pixel-level masks described by text, eliminating the need for manual thresholding and enabling universal prompts for open-set anomaly types. To support this paradigm, the authors introduce the MVTec-Ref dataset, containing 2,110 image-language-label triplets focused on small, anomaly-pattern-rich regions. They propose the Dual Query Token with Mask Group Transformer (DQFormer) augmented by Language-Gated Multi-Level Aggregation (LMA) to achieve efficient visual-textual fusion and robust multi-scale segmentation, using only Anomaly and Background tokens. Across experiments on MVTec-Ref, DQFormer outperforms state-of-the-art RIS baselines and demonstrates strong ablations, highlighting the potential of text-guided anomaly segmentation to push Industrial Anomaly Detection toward open-set capabilities with a unified model.

Abstract

Industrial Anomaly Detection (IAD) is vital for manufacturing, yet traditional methods face significant challenges: unsupervised approaches yield rough localizations requiring manual thresholds, while supervised methods overfit due to scarce, imbalanced data. Both suffer from the "One Anomaly Class, One Model" limitation. To address this, we propose Referring Industrial Anomaly Segmentation (RIAS), a paradigm leveraging language to guide detection. RIAS generates precise masks from text descriptions without manual thresholds and uses universal prompts to detect diverse anomalies with a single model. We introduce the MVTec-Ref dataset to support this, designed with diverse referring expressions and focusing on anomaly patterns, notably with 95% small anomalies. We also propose the Dual Query Token with Mask Group Transformer (DQFormer) benchmark, enhanced by Language-Gated Multi-Level Aggregation (LMA) to improve multi-scale segmentation. Unlike traditional methods using redundant queries, DQFormer employs only "Anomaly" and "Background" tokens for efficient visual-textual integration. Experiments demonstrate RIAS's effectiveness in advancing IAD toward open-set capabilities. Code: https://github.com/swagger-coder/RIAS-MVTec-Ref.

Figures (8)

• Figure 1: Comparison between traditional IAD and our RIAS. Traditional IAD methods generate coarse anomaly scores for localization, which also requires manual threshold adjustments to obtain the final classification result. Additionally, these methods suffer from the "One Anomaly Class, One Model" issue. In contrast, RIAS produces precise, fine-grained masks based on flexible textual descriptions, eliminating the need for manual thresholds. RIAS also supports the use of universal prompts to detect any anomaly types within one single unified model.
• Figure 1: Definitions of (a) orientation for textured objects and (b) position in the screw.
• Figure 2: Examples from MvTec AD to MVTec-Ref.
• Figure 2: Analysis of the dataset.
• Figure 3: The framework of DQFormer includes the overall framework, a Language-Gated Multi-Level Aggregation (LMA) module, and a Mask Group Transformer Layer. Initially, the Local-Enhanced Vision-Language Encoder (LVLE) takes $F^i_v$ and $F_l$ as inputs to the LMA, generating local-aware and language-aware features $F^i_{vl}$. These $F^i_{vl}$ features are then fused into the Vision Transformer through a gating mechanism. After a simple aggregation, they are fed into the Mask Group Decoder, where the Dual Query tokens $A$ and $B$ interact with language features $F_l$ and aggregated visual features $F^i_a$ sequentially. Within each Mask Group Transformer Layer, only $A$ interacts with the language features through a Masked Self-Attention mechanism. Finally, $A$ is multiplied by $F^3_a$ to generate the final mask output $M$, while $B$ is mapped to a No-Anomaly Identifier $E$.