Enhancing Out-of-Distribution Detection with Extended Logit Normalization

TL;DR

This work tackles the limited generalizability of LogitNorm for OOD detection by identifying feature-space collapse as a key limitation. It introduces Extended Logit Normalization (ELogitNorm), a hyperparameter-free, distance-aware training objective that uses the average distance to decision boundaries to scale logits, thereby preserving informative feature structure and improving OOD separability. Empirical results across CIFAR-10/100 and ImageNet benchmarks show that ELogitNorm consistently enhances a wide range of post-hoc OOD scores, yields better calibration, and maintains or improves ID accuracy compared to LogitNorm and other training-based approaches. The method is simple to integrate, robust to near- and far-OOD scenarios, and compatible with existing post-hoc scoring techniques, offering a practical advancement for reliable OOD detection in real-world systems.

Abstract

Out-of-distribution (OOD) detection is essential for the safe deployment of machine learning models. Recent advances have explored improved classification losses and representation learning strategies to enhance OOD detection. However, these methods are often tailored to specific post-hoc detection techniques, limiting their generalizability. In this work, we identify a critical issue in Logit Normalization (LogitNorm), which inhibits its effectiveness in improving certain post-hoc OOD detection methods. To address this, we propose Extended Logit Normalization ($\textbf{ELogitNorm}$), a novel hyperparameter-free formulation that significantly benefits a wide range of post-hoc detection methods. By incorporating feature distance-awareness to LogitNorm, $\textbf{ELogitNorm}$ shows more robust OOD separability and in-distribution (ID) confidence calibration than its predecessor. Extensive experiments across standard benchmarks demonstrate that our approach outperforms state-of-the-art training-time methods in OOD detection while maintaining strong ID classification accuracy.

Figures (7)

• Figure 1: Effectiveness of ELogitNorm applied with 4 Baseline OOD Methods. The OOD performance, as measured by AUROC $\%$ (higher is better) and FPR95 $\%$ (lower is better). The in-distribution dataset is CIFAR-10 and the scores have been averaged over four far-OOD datasets in the OpenOOD benchmark zhang2023openood.
• Figure 2: (a): The singular value spectrum of the covariance of training ID embedding $\mathbf{z}$ under different training losses including Cross-Entropy, LogitNormlogitnorm, and ELogitNorm (Ours). The ID data is CIFAR-10 trained on ResNet-18. (b): We train a ResNet18 on CIFAR-10 and set feature $\mathbf{z \in \mathbb{R}^2}$ before the last penultimate layer for visualization. (c) $||\mathbf{z}||$ and $||\mathbf{f}||$ denote the feature norm and logit norm, respectively. (d): Distance to origin $||\mathbf{z}||$ and distance to decision boundary $\mathcal{D}(\mathbf{z}, b)$, where $b$ is the corresponding decision boundary.
• Figure 3: Average far-OOD and near-OOD performance with 4 post-hoc methods MSP DanMSP, ReAct ReAct, GEN xixi2023GEN and SCALE xu2024scaling. ResNet18 are trained by Cross-Entropy, LogitNormlogitnorm, and ELogitNorm (Ours), respectively. ID data are CIFAR-10 and CIFAR-100.
• Figure 4: Validation accuracy during training. ResNet18 trained on CIFAR-10 over 100 training epochs. The proposed ELogitNorm (Ours) achieves faster convergence compared to LogitNormlogitnorm in the early training phase, while maintaining competitive final accuracy as Cross-Entropy .
• Figure 5: ID Accuracy. Comparison of Cross-Entropy, LogitNorm logitnorm and ELogitNorm (Ours) on CIFAR-10, CIFAR-100 and ImageNet-200 datasets.

Theorems & Definitions (2)

• Proposition 3.1
• Proposition 3.2