Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications

TL;DR

This paper tackles the challenge of trustworthy AI by presenting MACS, a post-hoc framework that unifies confidence estimation, out-of-distribution detection, and adversarial attack detection through multi-layer activation analysis. It builds compact corevectors, forms layer-wise classification-maps, and learns proto-maps from high-confidence, correctly classified samples to produce a cosine-based confidence score that reflects the coherence of the model's internal decision process. Empirically, MACS competes with or surpasses state-of-the-art baselines across CIFAR-100, CIFAR-100C, SVHN, and Places365 on both in-distribution and distribution-shift scenarios, while imposing lower online computational overhead. The approach demonstrates robustness to various attack strategies and input conditions, highlighting the practical value of internal activation signatures for real-world safety-critical deployments.

Abstract

The recent explosive growth in Deep Neural Networks applications raises concerns about the black-box usage of such models, with limited trasparency and trustworthiness in high-stakes domains, which have been crystallized as regulatory requirements such as the European Union Artificial Intelligence Act. While models with embedded confidence metrics have been proposed, such approaches cannot be applied to already existing models without retraining, limiting their broad application. On the other hand, post-hoc methods, which evaluate pre-trained models, focus on solving problems related to improving the confidence in the model's predictions, and detecting Out-Of-Distribution or Adversarial Attacks samples as independent applications. To tackle the limited applicability of already existing methods, we introduce Multi-Layer Analysis for Confidence Scoring (MACS), a unified post-hoc framework that analyzes intermediate activations to produce classification-maps. From the classification-maps, we derive a score applicable for confidence estimation, detecting distributional shifts and adversarial attacks, unifying the three problems in a common framework, and achieving performances that surpass the state-of-the-art approaches in our experiments with the VGG16 and ViTb16 models with a fraction of their computational overhead.

Figures (5)

• Figure 1: 's overview. The intermediate activations are processed, leading to a compact representation of the model's decision process ($\bm{{G}}$), which is compared with typical values for the predicted class ($\bm{{P}}$) resulting in a confidence score ($s\xspace$) that can be used for detecting misclassified samples, inputs or .
• Figure 2: Examples of $\bm{{P}}$ computed from VGG16 and ViTB16. For each network, we display the proto-maps with the largest and smallest Frobenius norm. Columns correspond to the indices of the analyzed layers, while rows represent the dataset classes.
• Figure 3: Confidence scores distribution for correctly and misclassified samples over VGG16 (left) and ViTB16 (right).
• Figure 4: Reliability diagrams for VGG16 (left) and ViTB16 (right).
• Figure 5: progression with the corruption intensity using samples from CIFAR-100's $\mathcal{D}^{\rm test}\xspace$ and corrupted samples from CIFAR-100C, with $cX$ indicating the corruption intensity, for VGG16 (left) and ViTB16 (right). As the intensity increases, scores decrease, making it easier to differentiate normal from corrupted samples.