Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Losses
Han Bao, Shinsaku Sakaue, Yuki Takezawa
TL;DR
This work analyzes gradient descent with arbitrary stepsize for linearly separable data under Fenchel–Young losses, showing convergence to an ε-optimal loss without requiring the self-bounding property when the loss has a separation margin. The key mechanism is a perceptron-style argument that tracks alignment with the separating direction, yielding rates T = Ω(ε^{-α}) where α depends on the potential φ generating the Fenchel–Young loss; in concrete instances, Tsallis entropy achieves Ω(ε^{-1/2}) and Rényi entropy achieves Ω(ε^{-1/3}) under margin. By instantiating with Shannon, Tsallis, and Rényi entropies, the paper demonstrates how the separation margin drives faster convergence than classical GD in the stable regime, while the logistic (Shannon) case recovers the standard rate when margin is absent. The results illuminate the role of margin over self-bounding for large-step GD, discuss stochastic extensions and implicit-bias implications, and lay out open problems including data-dependence and finite-time behavior.
Abstract
The gradient descent (GD) has been one of the most common optimizer in machine learning. In particular, the loss landscape of a neural network is typically sharpened during the initial phase of training, making the training dynamics hover on the edge of stability. This is beyond our standard understanding of GD convergence in the stable regime where arbitrarily chosen stepsize is sufficiently smaller than the edge of stability. Recently, Wu et al. (COLT2024) have showed that GD converges with arbitrary stepsize under linearly separable logistic regression. Although their analysis hinges on the self-bounding property of the logistic loss, which seems to be a cornerstone to establish a modified descent lemma, our pilot study shows that other loss functions without the self-bounding property can make GD converge with arbitrary stepsize. To further understand what property of a loss function matters in GD, we aim to show arbitrary-stepsize GD convergence for a general loss function based on the framework of \emph{Fenchel--Young losses}. We essentially leverage the classical perceptron argument to derive the convergence rate for achieving $ε$-optimal loss, which is possible for a majority of Fenchel--Young losses. Among typical loss functions, the Tsallis entropy achieves the GD convergence rate $T=Ω(ε^{-1/2})$, and the R{é}nyi entropy achieves the far better rate $T=Ω(ε^{-1/3})$. We argue that these better rate is possible because of \emph{separation margin} of loss functions, instead of the self-bounding property.