Multiclass Loss Geometry Matters for Generalization of Gradient Descent in Separable Classification
Matan Schliserman, Tomer Koren
TL;DR
This paper analyzes how unregularized Gradient Descent generalizes in separable multiclass linear classification by focusing on the geometry of the loss template rather than the decay rate alone. It introduces a template-based, $L_p$-smooth framework and proves finite-time population risk bounds that depend on the template's geometry (via $p$) and the loss tail $\rho$, with distinct behavior for $p=\infty$ versus $p=2$. A key technical contribution is a local Rademacher complexity bound for vector-valued predictors under $\rho$-tailed templates, together with a bound showing the optimal GD step-size grows with $p$, which yields improved convergence rates for larger $p$. The paper also establishes tight Euclidean-case lower bounds that formalize the unavoidable $k$-dependence when $p=2$, and provides applications to exponentially- and polynomially-tailed losses, including cross-entropy and general tail families. Overall, the results illuminate how multiclass loss geometry governs finite-time generalization, guiding loss-template choices in multiclass learning and advancing the understanding of implicit bias in gradient-based training.
Abstract
We study the generalization performance of unregularized gradient methods for separable linear classification. While previous work mostly deal with the binary case, we focus on the multiclass setting with $k$ classes and establish novel population risk bounds for Gradient Descent for loss functions that decay to zero. In this setting, we show risk bounds that reveal that convergence rates are crucially influenced by the geometry of the loss template, as formalized by Wang and Scott (2024), rather than of the loss function itself. Particularly, we establish risk upper bounds that holds for any decay rate of the loss whose template is smooth with respect to the $p$-norm. In the case of exponentially decaying losses, our results indicates a contrast between the $p=\infty$ case, where the risk exhibits a logarithmic dependence on $k$, and $p=2$ where the risk scales linearly with $k$. To establish this separation formally, we also prove a lower bound in the latter scenario, demonstrating that the polynomial dependence on $k$ is unavoidable. Central to our analysis is a novel bound on the Rademacher complexity of low-noise vector-valued linear predictors with a loss template smooth w.r.t.~general $p$-norms.