GCR: Geometry-Consistent Routing for Task-Agnostic Continual Anomaly Detection

TL;DR

This paper tackles task-agnostic continual anomaly detection with unknown category labels at test time, identifying cross-head routing instability as a major source of continual degradation. It introduces GCR, a geometry-consistent routing framework that decouples head selection from within-head anomaly scoring by operating in a shared frozen patch-embedding space and using per-category prototype banks constructed via coreset selection. Routing is performed by accumulating nearest-prototype distances across patches, after which anomaly maps are computed only within the routed head, enabling stable performance without end-to-end representation learning. Across MVTec AD and VisA, GCR achieves near-zero forgetting with competitive detection and localization, and ablations reveal routing strategy is the primary contributor to improvements, suggesting routing design can mitigate continual failures attributed to cross-head score misalignment. The work highlights a practical, training-free route to robust continual anomaly detection, with implications for scalable industrial inspection systems.

Abstract

Feature-based anomaly detection is widely adopted in industrial inspection due to the strong representational power of large pre-trained vision encoders. While most existing methods focus on improving within-category anomaly scoring, practical deployments increasingly require task-agnostic operation under continual category expansion, where the category identity is unknown at test time. In this setting, overall performance is often dominated by expert selection, namely routing an input to an appropriate normality model before any head-specific scoring is applied. However, routing rules that compare head-specific anomaly scores across independently constructed heads are unreliable in practice, as score distributions can differ substantially across categories in scale and tail behavior. We propose GCR, a lightweight mixture-of-experts framework for stabilizing task-agnostic continual anomaly detection through geometry-consistent routing. GCR routes each test image directly in a shared frozen patch-embedding space by minimizing an accumulated nearest-prototype distance to category-specific prototype banks, and then computes anomaly maps only within the routed expert using a standard prototype-based scoring rule. By separating cross-head decision making from within-head anomaly scoring, GCR avoids cross-head score comparability issues without requiring end-to-end representation learning. Experiments on MVTec AD and VisA show that geometry-consistent routing substantially improves routing stability and mitigates continual performance collapse, achieving near-zero forgetting while maintaining competitive detection and localization performance. These results indicate that many failures previously attributed to representation forgetting can instead be explained by decision-rule instability in cross-head routing. Code is available at https://github.com/jw-chae/GCR

Figures (2)

• Figure 1: Overview of GCR. (a) Training Phase) GCR extracts multi-layer patch features from a frozen Vision Transformer and concatenates them to form patch-level representations. For each category, a prototype memory bank is constructed via coreset selection to compactly represent normal patterns, without explicit parameter learning or anomaly scoring during training. (b) Inference Phase) Routing: Given a test image, the system first identifies the most suitable category by minimizing the accumulated nearest-prototype distance across all candidate memory banks, referred to as Geometry-Consistent Routing. Scoring: After routing, the anomaly map is computed solely within the selected expert using patch-wise distances to the retrieved prototypes. This explicit separation between routing and scoring ensures stable anomaly prediction under task-agnostic, multi-category, and continual settings.
• Figure 2: Qualitative anomaly localization results of GCR on MVTec AD. For each example, we show the input image (left) and the corresponding anomaly map (right), obtained by task-agnostic routing, optional top-$k$ head fusion, and top-$q$ pooling. GCR consistently highlights defect regions while suppressing irrelevant background responses across both object and texture categories, including subtle and spatially localized anomalies.