Enhancing Zero-Shot Anomaly Detection: CLIP-SAM Collaboration with Cascaded Prompts

TL;DR

The paper addresses zero-shot anomaly segmentation in data-scarce industrial settings by proposing a two-stage CLIP-SAM framework that leverages CLIP for anomaly localization and SAM for boundary-aware segmentation. It introduces two key modules: Co-Feature Point Prompt Generation (PPG), which generates positive and negative prompts from CLIP and SAM features to guide SAM toward anomalous regions, and Cascaded Prompts for SAM (CPS), which refines segmentation through a three-stage prompt cascade using a lightweight decoder. Empirical results on MVTec-AD and VisA demonstrate state-of-the-art performance, with notable gains on VisA in $F_1$-max ($+10.3 ext{\%}$) and $AP$ ($+7.7 ext{\%}$), while AUROC shows a modest trade-off. The work highlights the practical impact of jointly leveraging foundation models for anomaly detection, enabling robust segmentation with limited anomaly samples, and points to future work on making the approach more efficient for real-time deployment.

Abstract

Recently, the powerful generalization ability exhibited by foundation models has brought forth new solutions for zero-shot anomaly segmentation tasks. However, guiding these foundation models correctly to address downstream tasks remains a challenge. This paper proposes a novel two-stage framework, for zero-shot anomaly segmentation tasks in industrial anomaly detection. This framework excellently leverages the powerful anomaly localization capability of CLIP and the boundary perception ability of SAM.(1) To mitigate SAM's inclination towards object segmentation, we propose the Co-Feature Point Prompt Generation (PPG) module. This module collaboratively utilizes CLIP and SAM to generate positive and negative point prompts, guiding SAM to focus on segmenting anomalous regions rather than the entire object. (2) To further optimize SAM's segmentation results and mitigate rough boundaries and isolated noise, we introduce the Cascaded Prompts for SAM (CPS) module. This module employs hybrid prompts cascaded with a lightweight decoder of SAM, achieving precise segmentation of anomalous regions. Across multiple datasets, consistent experimental validation demonstrates that our approach achieves state-of-the-art zero-shot anomaly segmentation results. Particularly noteworthy is our performance on the Visa dataset, where we outperform the state-of-the-art methods by 10.3\% and 7.7\% in terms of {$F_1$-max} and AP metrics, respectively.

Figures (3)

• Figure 1: The CLIP-based method aligns text and image features, enabling precise anomaly localization but struggles to fully segment the entire anomaly area and its boundaries. On the other hand, the SAM-based approach successfully segments boundaries but often confuses normal and abnormal regions. Our method integrates the strengths of these two foundational models. Through the Co-Feature Point Prompt Generation (PPG) module, we generate initial point prompts by leveraging CLIP CLIP and SAM SAM. Subsequently, via the Cascaded Prompts for SAM (CPS) module, we further refine the mask quality by cascading hybrid prompts for SAM SAM, ultimately achieving successful and accurate anomaly segmentation with our framework.
• Figure 2: Comparison of visualization results among SAA+ SAA, APRIL-GAN APRIL-GAN, SDP+ SDP, Anomaly-CLIP Anomalyclip and ours on the MVTec-AD MVTec dataset and VisA visa dataset.
• Figure 3: Visualizations of SAM segmentation guided by the CPS module. When using point prompt Visualizations of SAM segmentation guided by the CPS module alone, the boundaries can be extremely blurry. With the addition of secondary points prompts and logit1, the delineation of abnormal boundaries becomes much clearer, although noise issues may persist. Upon introducing box prompt, the segmentation of boundaries can be achieved nearly perfectly.