One Model, Many Behaviors: Training-Induced Effects on Out-of-Distribution Detection

TL;DR

This work challenges the common assumption that higher in-distribution accuracy automatically yields better out-of-distribution detection. By fixing the ResNet-50 architecture and ImageNet data, it conducts a large-scale study across 56 models, 21 post-hoc OOD detectors, and eight OOD test sets, revealing a non-monotonic rise-then-fall relationship between ID accuracy and OOD performance. The results show strong interactions between training strategies and detectors, with no universally optimal method; generalist detectors based on geometry or higher-order statistics tend to be more robust across varied training recipes. The paper emphasizes the need for multi-model benchmarking that accounts for training-induced representation changes, and provides practical guidance favoring detector-model compatibility analysis over single-model evaluations.

Abstract

Out-of-distribution (OOD) detection is crucial for deploying robust and reliable machine-learning systems in open-world settings. Despite steady advances in OOD detectors, their interplay with modern training pipelines that maximize in-distribution (ID) accuracy and generalization remains under-explored. We investigate this link through a comprehensive empirical study. Fixing the architecture to the widely adopted ResNet-50, we benchmark 21 post-hoc, state-of-the-art OOD detection methods across 56 ImageNet-trained models obtained via diverse training strategies and evaluate them on eight OOD test sets. Contrary to the common assumption that higher ID accuracy implies better OOD detection performance, we uncover a non-monotonic relationship: OOD performance initially improves with accuracy but declines once advanced training recipes push accuracy beyond the baseline. Moreover, we observe a strong interdependence between training strategy, detector choice, and resulting OOD performance, indicating that no single method is universally optimal.

Figures (22)

• Figure 1: Top-1 classification accuracy is an unreliable indicator for OOD detection performance. This figure shows the relationship between in-distribution (ID) classification accuracy and out-of-distribution (OOD) detection performance for 56 ResNet-50 models, all sharing the same architecture but trained with different strategies. Each point represents one model and reports the mean AUROC (Area Under the Receiver Operating Characteristic Curve) over 21 OOD detection methods and eight OOD datasets. Color indicates the model's training category, while the marker shape uniquely identifies each model within that category.
• Figure 2: Relationship between ID accuracy and OOD detection performance (AUROC) across the OOD categories (near, far, extreme, synthetic). Each point represents one of the 56 ResNet-50 models, averaged over 21 OOD detection methods. Color indicates the model's training category, while the marker shape uniquely identifies each model within that category.
• Figure 3: Relationship between ID accuracy and OOD detection performance (AUROC) for each of the 21 OOD detection methods. Each point represents one of the 56 ResNet-50 models, averaged over eight OOD datasets. Color indicates the model's training category, while the marker shape uniquely identifies each model within that category. Best viewed on screen.
• Figure 4: Comparing the performance of MSP, KNN, and GRAM across multiple AUROC-based evaluation metrics. $\text{AUROC}_{\text{correct~vs.~incorrect}}$ evaluates failure prediction on ID data only, distinguishing between correctly and incorrectly classified samples. The remaining metrics assess OOD detection, either across all ID samples, only correctly classified ones, or only misclassified ones. Each boxplot shows the distribution over 56 models and four OOD categories.
• Figure 5: Relationship between ID classification accuracy and OOD detection performance. We distinguish between the ability of OOD detectors to separate correctly classified ID samples from OOD samples (left), and incorrectly classified ID samples from OOD samples (right). Each point represents one of the 56 ResNet-50 models, averaged over 21 OOD detection methods and four OOD categories. Color show the model's training category; marker shapes uniquely identify models within each category.