TIPS Over Tricks: Simple Prompts for Effective Zero-shot Anomaly Detection

TL;DR

This work tackles zero-shot anomaly detection in safety-critical domains where target-domain normal data are unavailable. By leveraging a spatially aware TIPS backbone and a decoupled prompting strategy, Tipsomaly achieves robust image-level detection and pixel-level localization without CLIP-centric tricks. The method introduces fixed prompts for image-level scoring and learnable prompts for localization, using two global tokens to integrate local evidence into the global decision, and demonstrates state-of-the-art performance across 14 industrial and medical datasets with meaningful gains in both image- and pixel-level metrics. The approach offers a lean, generalizable solution for cross-domain anomaly detection, with practical impact and openly available code.

Abstract

Anomaly detection identifies departures from expected behavior in safety-critical settings. When target-domain normal data are unavailable, zero-shot anomaly detection (ZSAD) leverages vision-language models (VLMs). However, CLIP's coarse image-text alignment limits both localization and detection due to (i) spatial misalignment and (ii) weak sensitivity to fine-grained anomalies; prior work compensates with complex auxiliary modules yet largely overlooks the choice of backbone. We revisit the backbone and use TIPS-a VLM trained with spatially aware objectives. While TIPS alleviates CLIP's issues, it exposes a distributional gap between global and local features. We address this with decoupled prompts-fixed for image-level detection and learnable for pixel-level localization-and by injecting local evidence into the global score. Without CLIP-specific tricks, our TIPS-based pipeline improves image-level performance by 1.1-3.9% and pixel-level by 1.5-6.9% across seven industrial datasets, delivering strong generalization with a lean architecture. Code is available at github.com/AlirezaSalehy/Tipsomaly.

Figures (8)

• Figure 1: Framework of Tipsomaly for zero-shot anomaly detection. An input image is encoded by the TIPS vision encoder $\Phi_v$ into dense patch features $Z^{\mathrm{M}}$ and two global tokens, $g_o^i$ (object-centric) and $g_s^i$ (spatial). We use decoupled prompting: the fixed detection prompt set $P_f$ yields fixed text prototypes $G_f$ for image-level decisions, while the learnable localization prompt set $P_l$ produces $G_l$ for pixel-level localization. The anomaly map $\hat{S}$ is computed via patch–text similarities between $Z^{\mathrm{M}}$ and $G_l$, then upsampled and smoothed to image resolution. For image-level detection, spatial token $g_s^i$ is scored against $G_f$, then the result is added to the strongest local evidence, $\max(\hat{S}_a)$, to obtain anomaly score $\hat{y}$.
• Figure 2: Qualitative results. Anomalies colored in original images. Tipsomaly yields cleaner localization with fewer false positives.
• Figure 3: Ablation over number of learnable tokens. The performance over various metrics and datasets suggests longer learnable prompts may overfit to the source domain and lose their ability to generalize effectively. Number of 8 learnable tokens yields the best overall results.
• Figure 4: Category-level localizations for the object capsule in the MVTec-AD. Ground-truth anomalous regions are outlined in green.
• Figure 5: Category-level localizations for the object hazelnut in the MVTec-AD. Ground-truth anomalous regions are outlined in green.