High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

TL;DR

The paper develops a unified framework for high-probability convergence of nonlinear streaming SGD under heavy-tailed noise, covering both non-convex and strongly convex costs with component-wise and joint nonlinearities. It proves rates near $\mathcal{O}(t^{-1/4+\epsilon})$ for non-convex losses and near $\mathcal{O}(t^{-1/2+\epsilon})$ for weighted averages in strongly convex settings, plus a last-iterate rate $\mathcal{O}(t^{-\zeta})$ with $\zeta\in(0,1)$ depending on problem parameters, nonlinearity, and noise. Crucially, the exponent $\zeta$ and the rates are shown to be independent of many problem specifics, and the analysis demonstrates that clipping is not always optimal, offering guidance on selecting nonlinearities in practice. The results extend high-probability guarantees to a broad class of nonlinearities and heavy-tailed noises, with both analytical and numerical support. Overall, the work provides a robust framework for designing streaming SGD methods that remain reliable under heavy-tailed disturbances while offering practical criteria for nonlinearity choice.

Abstract

We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-ζ} \right)$, where $ζ\in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $ζ$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $η\in (1,2]$, and establish convergence rates whose exponents go to zero as $η\rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Figures (2)

• Figure 1: Numerical verification of the inequality from Lemma \ref{['lm:analytical']}, for different values of $d$ and $B_0$. We can see that the inequality holds for $B_0$ finite or growing at an appropriate rate.
• Figure 2: Performance of sign, cclip and joint clipping for $d = 10$. Left to right: MSE performance and high-probability performance for $\varepsilon = \{0.1, 0.01 \}$, respectively.

Theorems & Definitions (34)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Example 1
• Example 2
• Example 3
• Definition 1
• Lemma 3.1
• Lemma 3.2