MRAD: Zero-Shot Anomaly Detection with Memory-Driven Retrieval

TL;DR

MRAD reframes zero-shot anomaly detection as feature–label retrieval over a memory bank instead of parametric fitting, combining a train-free base (MRAD-TF) with lightweight enhancements (MRAD-FT and MRAD-CLIP). A two-level memory bank stores image- and patch-level prototypes derived from an auxiliary dataset, enabling direct retrieval-based anomaly scores and segmentation. Across 16 industrial and medical datasets, MRAD variants consistently surpass state-of-the-art methods in both train-free and training-based settings, with MRAD-CLIP achieving top performance through dynamic, region-aware prompts. This retrieval-centric approach leverages the empirical data distribution for robust cross-domain generalization and efficient adaptation, and the authors provide code for reproducibility.

Abstract

Zero-shot anomaly detection (ZSAD) often leverages pretrained vision or vision-language models, but many existing methods use prompt learning or complex modeling to fit the data distribution, resulting in high training or inference cost and limited cross-domain stability. To address these limitations, we propose Memory-Retrieval Anomaly Detection method (MRAD), a unified framework that replaces parametric fitting with a direct memory retrieval. The train-free base model, MRAD-TF, freezes the CLIP image encoder and constructs a two-level memory bank (image-level and pixel-level) from auxiliary data, where feature-label pairs are explicitly stored as keys and values. During inference, anomaly scores are obtained directly by similarity retrieval over the memory bank. Based on the MRAD-TF, we further propose two lightweight variants as enhancements: (i) MRAD-FT fine-tunes the retrieval metric with two linear layers to enhance the discriminability between normal and anomaly; (ii) MRAD-CLIP injects the normal and anomalous region priors from the MRAD-FT as dynamic biases into CLIP's learnable text prompts, strengthening generalization to unseen categories. Across 16 industrial and medical datasets, the MRAD framework consistently demonstrates superior performance in anomaly classification and segmentation, under both train-free and training-based settings. Our work shows that fully leveraging the empirical distribution of raw data, rather than relying only on model fitting, can achieve stronger anomaly detection performance. The code will be publicly released at https://github.com/CROVO1026/MRAD.

Figures (37)

• Figure 1: Most existing ZSAD methods parameterize and fit $p(y\mid x)$ solely via a trainable model, which may cause information loss. Our MRAD is to directly access the empirical training distribution by retrieving from a feature-label memory bank. The dashed line denotes the inference process.
• Figure 2: Mean similarity scores for different query-key relations (queries from four datasets; keys and values from VisA). (a) Train-free setting: the consistent ordering $\mathrm{N}_{q}\mathrm{N}_{k} > \mathrm{A}_{q}\mathrm{N}_{k}$ and $\mathrm{A}_{q}\mathrm{A}_{k} > \mathrm{N}_{q}\mathrm{A}_{k}$ shows that similarity to normal/abnormal keys provides stable discriminative signals. (b) Fine-tune setting: a lightweight fine-tuning further enlarges the margin between $\mathrm{A}_{q}\mathrm{A}_{k}$ and $\mathrm{N}_{q}\mathrm{A}_{k}$, demonstrating that fine-tuning improves separability while preserving the same ordering.
• Figure 3: Overall architecture of the proposed MRAD framework. (a) MRAD-TF/FT: query features from a frozen CLIP image encoder are matched to a two-level feature-label memory bank; MRAD-TF uses direct similarity retrieval, while MRAD-FT adds linear layers into cross-atten module. (b) MRAD-CLIP: priors from MRAD-FT are injected as dynamic biases into learnable text prompts, enhancing cross-modal alignment and anomaly localization.
• Figure 4: Comparison of anomaly segmentation results between MRAD-CLIP and other methods.
• Figure 5: Ablation results of different MRAD variants. Radar charts on six industrial datasets compare image-level (I-AUROC/I-AP) and pixel-level (P-AUROC/PRO) metrics.