Towards Adversarial Robustness and Uncertainty Quantification in DINOv2-based Few-Shot Anomaly Detection

TL;DR

The paper addresses the vulnerability and trustworthiness of DINOv2-based few-shot anomaly detection (FSAD) by studying adversarial perturbations and calibration of anomaly scores. It introduces a white-box FGSM-type attack via a lightweight linear probe attached to frozen DINOv2 features, preserving the original k-NN detection while enabling gradient-based perturbations. Empirical results on MVTec-AD and VisA show substantial performance degradation under perturbations and poor initial calibration of anomaly scores, which are mitigated by a simple Platt scaling post-hoc that also increases predictive entropy for attacked inputs. The findings establish a practical framework for evaluating robustness and deploying uncertainty-aware FSAD systems, arguing that adversarial robustness and principled uncertainty quantification are essential for real-world trustworthiness.

Abstract

Foundation models such as DINOv2 have shown strong performance in few-shot anomaly detection, yet two key questions remain unexamined: (i) how susceptible are these detectors to adversarial perturbations; and (ii) how well do their anomaly scores reflect calibrated uncertainty? Building on AnomalyDINO, a training-free deep nearest-neighbor detector over DINOv2 features, we present one of the first systematic studies of adversarial attacks and uncertainty estimation in this setting. To enable white-box gradient attacks while preserving test-time behavior, we attach a lightweight linear head to frozen DINOv2 features only for crafting perturbations. Using this heuristic, we evaluate the impact of FGSM across the MVTec-AD and VisA datasets and observe consistent drops in F1, AUROC, AP, and G-mean, indicating that imperceptible perturbations can flip nearest-neighbor relations in feature space to induce confident misclassification. Complementing robustness, we probe reliability and find that raw anomaly scores are poorly calibrated, revealing a gap between confidence and correctness that limits safety-critical use. As a simple, strong baseline toward trustworthiness, we apply post-hoc Platt scaling to the anomaly scores for uncertainty estimation. The resulting calibrated posteriors yield significantly higher predictive entropy on adversarially perturbed inputs than on clean ones, enabling a practical flagging mechanism for attack detection while reducing calibration error (ECE). Our findings surface concrete vulnerabilities in DINOv2-based few-shot anomaly detectors and establish an evaluation protocol and baseline for robust, uncertainty-aware anomaly detection. We argue that adversarial robustness and principled uncertainty quantification are not optional add-ons but essential capabilities if anomaly detection systems are to be trustworthy and ready for real-world deployment.

Figures (5)

• Figure 1: (a) Feature space overview showing adversarial perturbations pushing samples across the decision boundary. (b) Clean and adversarial 'normal' capsule test images (left) with corresponding patch-wise distance score maps using 1-nearest neighbor matching (right). The clean image yields low, uniform distances, while the adversarial case shows higher and irregular scores.
• Figure 2: Illustration of the adversarial perturbation and detection pipeline. A clean query image $x_q$ is processed through a DINOv2 encoder, with patch-wise nearest neighbor (NN) matching against a patch bank producing the anomaly score $s(x_q)$. Simultaneously, a linear classifier trained with a loss $\mathcal{L}(x_q, y_q)$ provides gradients used to craft an adversarial example $x_q'$ via FGSM. The adversarial image is then passed through the same DINOv2 encoder and patch-wise NN matching, yielding the adversarial anomaly score $s(x_q')$, enabling robustness analysis of the detection system.
• Figure 3: Comparison of anomaly detection under clean ($x_q$) and adversarial ($x_q^{adv}$) inputs. Clean and adversarial images are processed through AnomalyDINO, $AD$ to produce uncalibrated anomaly scores, $s$, which are further calibrated using Platt scaling ($\hat{p}$). The predictive entropy distributions (with mean and standard deviation reported) illustrate how calibration affects the uncertainty estimates in both settings.
• Figure 4: Graphical calibration results for anomaly scores on VisA (left) and MVTec-AD (right). (a) Reliability diagrams compare uncalibrated scores and Platt-scaled probabilities against the diagonal of perfect calibration. (b) Histograms of predicted probabilities show the distribution shift induced by Platt scaling.
• Figure 5: Predictive entropy of platt-scaled anomaly scores for 1-shot setting across both datasets, averaged across all test samples under both clean and adversarial input conditions. Entropy was computed for each prediction, with higher values indicating greater uncertainty. The legend includes summary statistics of the histograms, i.e. mean and standard deviation.