Dissecting Mahalanobis: How Feature Geometry and Normalization Shape OOD Detection

TL;DR

This work investigates how the geometry of feature representations and normalization choices shape Mahalanobis-based OOD detection. Through an extensive empirical study over diverse foundation-model features, it shows that OOD performance is not universal and is strongly predicted by geometric properties such as spectral decay and intrinsic dimensionality, with a particular emphasis on the interplay between local manifold structure and cluster compactness. The authors introduce radially scaled $\ell_2$ normalization, parameterized by $\beta$, to actively reshape the feature space and improve OOD separation, and demonstrate a regression framework to predict an optimal $\beta$ using only in-distribution data. Together, these contributions offer a practical blueprint for designing more reliable OOD detectors by explicitly controlling representation geometry and normalization, with implications for deploying large-scale vision models in open-world settings.

Abstract

Out-of-distribution (OOD) detection is critical for the reliable deployment of deep learning models. hile Mahalanobis distance methods are widely used, the impact of representation geometry and normalization on their performance is not fully understood, which may limit their downstream application. To address this gap, we conducted a comprehensive empirical study across diverse image foundation models, datasets, and distance normalization schemes. First, our analysis shows that Mahalanobis-based methods aren't universally reliable. Second, we define the ideal geometry for data representations and demonstrate that spectral and intrinsic-dimensionality metrics can accurately predict a model's OOD performance. Finally, we analyze how normalization impacts OOD performance. Building upon these studies, we propose radially scaled $\ell_2$ normalization, a method that generalizes the standard $\ell_2$ normalization recently applied to Mahalanobis-based OOD detection. Our approach introduces a tunable parameter to directly control the radial geometry of the feature space, systematically contracting or expanding representations to significantly improve OOD detection performance. By bridging the gap between representation geometry, normalization, and OOD performance, our findings offer new insights into the design of more effective and reliable deep learning models.

Figures (15)

• Figure 1: Effect of $\beta$-scaled $\ell_2$ normalization effect 2D feature geometry and Mahalanobis decision surfaces. Optimal $\beta$ improve OOD detection performance. Gray arrows indicate the mapping from the original to the transformed space. Increasing $\beta$ contracts norms and tightens clusters, yielding smoother, more localized decision regions; negative $\beta$ spreads points and broadens regions. Choosing an appropriate $\beta$ improves class separation and downstream OOD detection.
• Figure 2: OOD detection performance across model families on the NINCO. RMD consistently outperforms the standard MD, especially for models pretrained but not fine-tuned on ImageNet.
• Figure 3: Dimension-wise analysis of OOD separation (top) and FPR under progressive dimension ablation (bottom). Large embedding-space separation does not necessarily guarantee superior detection.
• Figure 4: Spectral ratios across models. Higher ratios indicate richer within-class variation and more expressive feature spaces. Fine-tuning tends to increase $S_b$ while preserving $S_w$.
• Figure 5: Spearman correlations between representation metrics and OOD performance across Mahalanobis variants. The three Mahalanobis-based detectors exploit different geometric cues, which explains their distinct correlation patterns. These correlations also hold for Pearson correlations (see Appendix \ref{['app:pearson-correlation']}).