Understanding Softmax Confidence and Uncertainty

TL;DR

This work analyzes why softmax confidence often correlates with epistemic uncertainty in standard OOD detection tasks and identifies two implicit biases that drive this behavior: (i) an approximately optimal final-layer decision boundary structure and (ii) a depth-enabled filtering of task-specific features. It formalizes the notion of a valid OOD region for common uncertainty estimators, and provides theoretical and empirical support for the role of boundary geometry and feature filtering in improving OOD detection. Diagnostics reveal that overlap between training and OOD representations in the final layer largely explains softmax failures, while pre-training or fine-tuning reduces this overlap and substantially improves reliability. The findings offer a practical reframing of softmax-based uncertainty, highlight the limits of low-dimensional intuition, and point to design strategies—such as promoting bijective or more diverse final-layer representations and leveraging pre-training—to enhance uncertainty estimation in real-world systems.

Abstract

It is often remarked that neural networks fail to increase their uncertainty when predicting on data far from the training distribution. Yet naively using softmax confidence as a proxy for uncertainty achieves modest success in tasks exclusively testing for this, e.g., out-of-distribution (OOD) detection. This paper investigates this contradiction, identifying two implicit biases that do encourage softmax confidence to correlate with epistemic uncertainty: 1) Approximately optimal decision boundary structure, and 2) Filtering effects of deep networks. It describes why low-dimensional intuitions about softmax confidence are misleading. Diagnostic experiments quantify reasons softmax confidence can fail, finding that extrapolations are less to blame than overlap between training and OOD data in final-layer representations. Pre-trained/fine-tuned networks reduce this overlap.

Figures (17)

• Figure 1: a) In low-dimensions, softmax confidence is unreliable as a measure of uncertainty -- OOD data far from the training distribution falls into high confidence extrapolation regions. However, with more complex tasks and deeper networks, implicit biases encourage higher uncertainty for data outside the training distribution. b) For a LeNet trained on three classes of MNIST, OOD data from Fashion MNIST is reliably mapped to the low confidence region of the softmax layer.
• Figure 2: The valid OOD region is shown in green for each uncertainty estimator. Gaussian ellipses capture 95% of clusters. For softmax confidence, OOD data must fall closer to a decision boundary than 95% of training data, whilst the density estimator only demands it fall outside of training clusters.
• Figure 3: Optimal decision boundary structure for $H=2, K=3$. The valid OOD region is overlaid in green.
• Figure 4: Histograms of final-layer weight properties for various architectures/datasets show reasonable agreement with the theoretically optimal structure.
• Figure 5: Illustration of various softmax structures, along with results when $\mathcal{D}_\text{in}$ is $K=3$ classes of MNIST, and $\mathcal{D}_\text{out}$ is Fashion MNIST. Whilst all structures achieve similar test accuracy, AUROC is largest when the softmax weight vectors are either trainable, or fixed in an optimal configuration. Fig. \ref{['fig_weird_structures_pca']} provides PCA visualisations of the trained networks. Mean $\pm$ 1 std. err. over three runs.

Theorems & Definitions (42)

• Definition 1
• Definition 2
• Proposition 1
• proof
• Remark
• Theorem 1
• Remark
• proof
• Lemma 1.1
• proof