Similarity-Distance-Magnitude Activations

TL;DR

The paper introduces the Similarity-Distance-Magnitude (SDM) activation as a robust, interpretable replacement for softmax, integrating exemplar-based Similarity, Distance to training, and Magnitude (decision-boundary) signals. An SDM adaptor computes these signals from dense representations, enabling a data-driven SDM loss and training regime, plus an SDM estimator that partitions predictions via class-conditioned eCDFs to control accuracy under selective classification. Across binary sentiment and fact-check tasks, SDM-based estimators demonstrate superior reliability and robustness to covariate shifts and out-of-distribution inputs compared with traditional calibration methods, while preserving informative in-distribution behavior. The work also provides an algorithm to determine a High-Reliability region for admissions, linking calibration, conformal ideas, and exemplar-based interpretability for practical decision pipelines. Overall, SDM offers a principled framework for uncertainty estimation, selective classification, and OOD detection in large language models with interpretable exemplar guidance.

Abstract

We introduce the Similarity-Distance-Magnitude (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding Similarity (i.e., correctly predicted depth-matches into training) awareness and Distance-to-training-distribution awareness to the existing output Magnitude (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching. We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class- and prediction-conditional accuracy among selective classifications. When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to co-variate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.