Is Softmax Loss All You Need? A Principled Analysis of Softmax-family Loss
Yuanhao Pu, Defu Lian, Enhong Chen
TL;DR
This work investigates whether Softmax loss is sufficient for large-class classification and ranking by embedding Softmax-family surrogates into the Fenchel–Young framework. It formalizes classification and ranking calibration via Bregman divergences, derives consistency guarantees for both dense (Softmax) and sparse (Sparsemax, $\alpha$-Entmax, Rankmax) surrogates, and analyzes optimization dynamics through Jacobian-based smoothness. A unified bias–variance decomposition for approximate surrogates is proposed, enabling concrete trade-offs between statistical fidelity and efficiency. Extensive experiments on recommender-system tasks validate the theory, showing how per-epoch complexity and bias–variance considerations guide loss selection in extreme-class settings and offering practical guidance for large-scale applications.
Abstract
The Softmax loss is one of the most widely employed surrogate objectives for classification and ranking tasks. To elucidate its theoretical properties, the Fenchel-Young framework situates it as a canonical instance within a broad family of surrogates. Concurrently, another line of research has addressed scalability when the number of classes is exceedingly large, in which numerous approximations have been proposed to retain the benefits of the exact objective while improving efficiency. Building on these two perspectives, we present a principled investigation of the Softmax-family losses. We examine whether different surrogates achieve consistency with classification and ranking metrics, and analyze their gradient dynamics to reveal distinct convergence behaviors. We also introduce a systematic bias-variance decomposition for approximate methods that provides convergence guarantees, and further derive a per-epoch complexity analysis, showing explicit trade-offs between effectiveness and efficiency. Extensive experiments on a representative task demonstrate a strong alignment between consistency, convergence, and empirical performance. Together, these results establish a principled foundation and offer practical guidance for loss selections in large-class machine learning applications.