Training Free Zero-Shot Visual Anomaly Localization via Diffusion Inversion

TL;DR

DIVAD addresses zero-shot visual anomaly detection without any normal training data or task-specific prompts. It exploits diffusion inversion of a pretrained Stable Diffusion model to reconstruct a normal-looking image and detects anomalies from pixel- and patch-level feature discrepancies, aided by a self-supervised backbone (DINO) and an object mask refinement (CutLER). The approach is training-free and category-agnostic, relying solely on a generic text description and latent-space inversion. Empirical results on VISA, MVTec-AD, and MPDD show state-of-the-art or competitive pixel-level localization for training-free ZSAD, underscoring the potential of diffusion priors for prompt-free anomaly localization.

Abstract

Zero-Shot image Anomaly Detection (ZSAD) aims to detect and localise anomalies without access to any normal training samples of the target data. While recent ZSAD approaches leverage additional modalities such as language to generate fine-grained prompts for localisation, vision-only methods remain limited to image-level classification, lacking spatial precision. In this work, we introduce a simple yet effective training-free vision-only ZSAD framework that circumvents the need for fine-grained prompts by leveraging the inversion of a pretrained Denoising Diffusion Implicit Model (DDIM). Specifically, given an input image and a generic text description (e.g., "an image of an [object class]"), we invert the image to obtain latent representations and initiate the denoising process from a fixed intermediate timestep to reconstruct the image. Since the underlying diffusion model is trained solely on normal data, this process yields a normal-looking reconstruction. The discrepancy between the input image and the reconstructed one highlights potential anomalies. Our method achieves state-of-the-art performance on VISA dataset, demonstrating strong localisation capabilities without auxiliary modalities and facilitating a shift away from prompt dependence for zero-shot anomaly detection research. Code is available at https://github.com/giddyyupp/DIVAD.

Figures (7)

• Figure 1: An overview of the current training-free ZSAD methods vs DIVAD. Current methods utilize pretrained large Vision Foundation Models (VFM) with specially curated object specific guided-prompts. Our method is free from any prompt tuning, and makes use of pretrained Stable Diffusion (SD) model to first invert a given image, and then reconstruct it through denoising.
• Figure 2: The processing pipeline of DIVAD. Given test image first encoded to latent representation, the SD UNet ldm predicts the noise $\epsilon_T$ for $T$ timesteps using DDIM Inversion process. Predicted noise is added to the input image latent $z$ to obtain inverted latent $z_T$. Then, this inverted latent is fed to the SD UNet model for denoising. Denoised latent is forwarded to the image decoder to recover a clean normal image. Instead of pixel wise comparison of input and reconstructed images which generates noisy regions, we perform segmentation on feature-level by utilizing a self-supervised model. Additionally, we make use of an object segmenter (CutLER wang2023cut) to improve the segmentation performance.
• Figure 3: Qualitative results of DIVAD on MVTec-AD (top) and VISA (bottom) for selected classes. DIVAD successfully detects the anomaly locations on various object and texture classes.
• Figure 4: Examples of DIVAD predictions on the MPDD dataset.
• Figure 5: Diffusion inversion visuals for an object on the top and a texture on the bottom. Starting from a small $T'$ captures more features from the input image as expected. A value between $10$ and $15$ achieves the best reconstruction.