Is Training Necessary for Anomaly Detection?

TL;DR

The paper questions the necessity of task-specific training for multi-class unsupervised anomaly detection (MUAD) by identifying a fidelity–stability trade-off in reconstruction-based methods. It introduces RAD, a training-free, memory-based retrieval framework that uses a frozen encoder and a multi-layer anomaly-free feature memory to detect anomalies via global-then-patch retrieval and spatially conditioned scoring. The authors prove that retrieval-based anomaly scores upper-bound reconstruction residuals and demonstrate state-of-the-art performance on four benchmarks, including robust few-shot and cold-start results. This work suggests that, with high-quality frozen representations and memory-based retrieval, competitive MUAD is achievable without training, potentially reshaping future anomaly detection approaches and datasets.

Abstract

Current state-of-the-art multi-class unsupervised anomaly detection (MUAD) methods rely on training encoder-decoder models to reconstruct anomaly-free features. We first show these approaches have an inherent fidelity-stability dilemma in how they detect anomalies via reconstruction residuals. We then abandon the reconstruction paradigm entirely and propose Retrieval-based Anomaly Detection (RAD). RAD is a training-free approach that stores anomaly-free features in a memory and detects anomalies through multi-level retrieval, matching test patches against the memory. Experiments demonstrate that RAD achieves state-of-the-art performance across four established benchmarks (MVTec-AD, VisA, Real-IAD, 3D-ADAM) under both standard and few-shot settings. On MVTec-AD, RAD reaches 96.7\% Pixel AUROC with just a single anomaly-free image compared to 98.5\% of RAD's full-data performance. We further prove that retrieval-based scores theoretically upper-bound reconstruction-residual scores. Collectively, these findings overturn the assumption that MUAD requires task-specific training, showing that state-of-the-art anomaly detection is feasible with memory-based retrieval. Our code is available at https://github.com/longkukuhi/RAD.

Figures (7)

• Figure 1: Visualizations of MUAD with only one reference by RAD.
• Figure 2: Overview of the proposed RAD framework.
• Figure 3: Few-shot results on MVTec-AD. $\dag$ indicates few-shot-specific method.
• Figure A1: Anomaly maps visualization on 3D-ADAM. All samples are randomly chosen.
• Figure A2: Anomaly maps visualization on MVTec-AD. All samples are randomly chosen.

Theorems & Definitions (11)

• Lemma 2.2: Decoder amplification
• Lemma 4.1: Non-expansiveness
• Lemma 3.1: singular value inequality
• proof
• Remark 3.2: Compatibility with strong compression
• Proposition 4.1: Empirical saturation of retrieval
• proof
• Proposition 4.2: Canonical $1$-Lipschitz extension
• proof
• Lemma 4.3: Non-expansiveness, restated