AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors

TL;DR

This work tackles zero-shot anomaly detection with purely visual foundation models (VFMs) by addressing two bottlenecks: limited auxiliary-data diversity and shallow VFM adaptation. It introduces AnomalyVFM, a framework that combines a three-stage synthetic dataset generator with a parameter-efficient backbone adaptation using low-rank feature adapters and a confidence-weighted loss to produce precise pixel-level anomaly maps and image-level scores from VFMs. Across nine industrial and nine medical datasets, AnomalyVFM achieves a 94.1% average image-level AUROC and outperforms prior zero-shot methods by 3.3 percentage points, while also generalising to medical data and matching few-shot SOTA when lightly finetuned. The approach offers a practical, model-agnostic path to deploy robust zero-shot anomaly detection by transformingVFMs into capable detectors without large-scale in-domain data or heavy fine-tuning.

Abstract

Zero-shot anomaly detection aims to detect and localise abnormal regions in the image without access to any in-domain training images. While recent approaches leverage vision-language models (VLMs), such as CLIP, to transfer high-level concept knowledge, methods based on purely vision foundation models (VFMs), like DINOv2, have lagged behind in performance. We argue that this gap stems from two practical issues: (i) limited diversity in existing auxiliary anomaly detection datasets and (ii) overly shallow VFM adaptation strategies. To address both challenges, we propose AnomalyVFM, a general and effective framework that turns any pretrained VFM into a strong zero-shot anomaly detector. Our approach combines a robust three-stage synthetic dataset generation scheme with a parameter-efficient adaptation mechanism, utilising low-rank feature adapters and a confidence-weighted pixel loss. Together, these components enable modern VFMs to substantially outperform current state-of-the-art methods. More specifically, with RADIO as a backbone, AnomalyVFM achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points. Project Page: https://maticfuc.github.io/anomaly_vfm/

Figures (11)

• Figure 1: Vision–language models excel in zero-shot anomaly detection thanks to their high-level concept knowledge, but purely visual foundation models hold untapped potential. AnomalyVFM unlocks this potential by addressing the two practical limitations that hinder VFM underperformance: suboptimal training sets and suboptimal fine-tuning procedures.
• Figure 1: Failure Cases in Image Generation Process
• Figure 2: Examples of generated anomaly-free images $I$, anomalous images $I_a$ and corresponding masks $M$.
• Figure 2: Anomalous Area Distribution in the generated synthetic dataset.
• Figure 3: Dataset generation pipeline. The image $I$ is generated using a text-conditioned image generation model. Then, the foreground mask $M_{fg}$ is extracted and an anomalous region $R$ is sampled from it. Then, the anomalous image $I_a$ is generated by inpainting an anomaly inside $R$. Finally, features are extracted from $I$ and $I_a$, and then compared and thresholded to obtain $M$.