Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models

Zhen Qu, Xian Tao, Xiaoyi Bao, Dingrong Wang, ShiChen Qu, Zhengtao Zhang, Xingang Wang

TL;DR

AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens, establishing a unified anchor-guided segmentation paradigm for zero-shot visual anomaly segmentation.

Abstract

Large multimodal models (LMMs) exhibit strong task generalization capabilities, offering new opportunities for zero-shot visual anomaly segmentation (ZSAS). However, existing LMM-based segmentation approaches still face fundamental limitations: anomaly concepts are inherently abstract and context-dependent, lacking stable visual prototypes, and the weak alignment between high-level semantic embeddings and pixel-level spatial features hinders precise anomaly localization. To address these challenges, we present AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens-[SEG], [NOR], and [ANO], establishing a unified anchor-guided segmentation paradigm. Specifically, [SEG] serves as an absolute semantic anchor that translates abstract anomaly semantics into explicit, spatially grounded visual entities (e.g., holes or scratches), while [NOR] and [ANO] act as relative anchors that model the contextual contrast between normal and abnormal patterns across categories. To further enhance cross-modal alignment, we introduce a Semantic-Pixel Alignment Module (SPAM) that aligns language-level semantic embeddings with high-resolution visual features, along with an Anchor-Guided Mask Decoder (AGMD) that performs anchor-conditioned mask prediction for precise anomaly localization. In addition, we curate Anomaly-Instruct20K, a large-scale instruction dataset that organizes anomaly knowledge into structured descriptions of appearance, shape, and spatial attributes, facilitating effective learning and integration of the proposed semantic anchors. Extensive experiments on six industrial and medical benchmarks demonstrate that AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.

AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models

TL;DR

AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens, establishing a unified anchor-guided segmentation paradigm for zero-shot visual anomaly segmentation.

Abstract

Large multimodal models (LMMs) exhibit strong task generalization capabilities, offering new opportunities for zero-shot visual anomaly segmentation (ZSAS). However, existing LMM-based segmentation approaches still face fundamental limitations: anomaly concepts are inherently abstract and context-dependent, lacking stable visual prototypes, and the weak alignment between high-level semantic embeddings and pixel-level spatial features hinders precise anomaly localization. To address these challenges, we present AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens-[SEG], [NOR], and [ANO], establishing a unified anchor-guided segmentation paradigm. Specifically, [SEG] serves as an absolute semantic anchor that translates abstract anomaly semantics into explicit, spatially grounded visual entities (e.g., holes or scratches), while [NOR] and [ANO] act as relative anchors that model the contextual contrast between normal and abnormal patterns across categories. To further enhance cross-modal alignment, we introduce a Semantic-Pixel Alignment Module (SPAM) that aligns language-level semantic embeddings with high-resolution visual features, along with an Anchor-Guided Mask Decoder (AGMD) that performs anchor-conditioned mask prediction for precise anomaly localization. In addition, we curate Anomaly-Instruct20K, a large-scale instruction dataset that organizes anomaly knowledge into structured descriptions of appearance, shape, and spatial attributes, facilitating effective learning and integration of the proposed semantic anchors. Extensive experiments on six industrial and medical benchmarks demonstrate that AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.
Paper Structure (13 sections, 11 equations, 4 figures, 4 tables)

This paper contains 13 sections, 11 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Multimodal Models
  4. Zero-Shot Anomaly Segmentation
  5. Method
  6. Model Design
  7. Training Objectives
  8. Anomaly-Instruct20K Dataset
  9. Experiment
  10. Experimental Setup
  11. Comparison with State-of-the-art methods
  12. Ablation
  13. Conclusion

Figures (4)

  • Figure 1: Comparison of zero-shot anomaly segmentation capabilities between LISA LISA and our AG-VAS.
  • Figure 2: AG-VAS is an effective and efficient LMM for zero-shot anomaly segmentation and understanding. (Left: raw image; Right: segmentation result.) The visualization results are shown across the following scenarios: (1)–(4) direct segmentation, (5)–(6) describe-then-segment, (7) segment-then-explain, and (8) segmentation within dialogue.
  • Figure 3: The pipeline of AG-VAS. We design two types of semantic anchors: the absolute anchor [SEG], which injects world and contextual knowledge about the appearance, structure, and location of anomalies (e.g., holes, scratches) into the segmentation process, and the relative normal/anomalous anchors [NOR]/[ANO], which guide the model to understand anomalies in relation to normal patterns. All parameters except those for the semantic/pixel image encoder, visual projector and LLM are trainable.
  • Figure 4: (a) An example from our Anomaly-Instruct20K dataset. (b) An example of instruction-tuning with dynamic fusion of templates during the actual training process. Note that only one specific template is shown here, with further details provided in the Appendix.