Clipped Gradient Methods for Nonsmooth Convex Optimization under Heavy-Tailed Noise: A Refined Analysis
Zijian Liu
TL;DR
This work analyzes Clipped SGD for nonsmooth convex optimization under heavy-tailed gradient noise with ${\frak p}\in(1,2]$, introducing a generalized effective dimension ${d_{\rm eff}}=\sigma_{\rm l}^2/\sigma_{\rm s}^2$ to capture noise structure. It provides refined high-probability rates that beat prior bounds by leveraging a tighter application of Freedman’s inequality and finer clipping-error bounds, and extends the analysis to faster convergence in expectation, producing new lower bounds that nearly match the upper bounds. A Stabilized Clipped SGD variant achieves anytime convergence without knowing the horizon, further improving practical robustness. The paper also establishes new high-probability and in-expectation lower bounds, demonstrating the optimality of the refined analysis in expectation and highlighting structural advantages conferred by the generalized effective dimension. These results collectively advance understanding of first-order methods under heavy-tailed noise and offer concrete guidance for achieving fast, reliable convergence in practice.
Abstract
Optimization under heavy-tailed noise has become popular recently, since it better fits many modern machine learning tasks, as captured by empirical observations. Concretely, instead of a finite second moment on gradient noise, a bounded ${\frak p}$-th moment where ${\frak p}\in(1,2]$ has been recognized to be more realistic (say being upper bounded by $σ_{\frak l}^{\frak p}$ for some $σ_{\frak l}\ge0$). A simple yet effective operation, gradient clipping, is known to handle this new challenge successfully. Specifically, Clipped Stochastic Gradient Descent (Clipped SGD) guarantees a high-probability rate ${\cal O}(σ_{\frak l}\ln(1/δ)T^{1/{\frak p}-1})$ (resp. ${\cal O}(σ_{\frak l}^2\ln^2(1/δ)T^{2/{\frak p}-2})$) for nonsmooth convex (resp. strongly convex) problems, where $δ\in(0,1]$ is the failure probability and $T\in\mathbb{N}$ is the time horizon. In this work, we provide a refined analysis for Clipped SGD and offer two faster rates, ${\cal O}(σ_{\frak l}d_{\rm eff}^{-1/2{\frak p}}\ln^{1-1/{\frak p}}(1/δ)T^{1/{\frak p}-1})$ and ${\cal O}(σ_{\frak l}^2d_{\rm eff}^{-1/{\frak p}}\ln^{2-2/{\frak p}}(1/δ)T^{2/{\frak p}-2})$, than the aforementioned best results, where $d_{\rm eff}\ge1$ is a quantity we call the $\textit{generalized effective dimension}$. Our analysis improves upon the existing approach on two sides: better utilization of Freedman's inequality and finer bounds for clipping error under heavy-tailed noise. In addition, we extend the refined analysis to convergence in expectation and obtain new rates that break the known lower bounds. Lastly, to complement the study, we establish new lower bounds for both high-probability and in-expectation convergence. Notably, the in-expectation lower bounds match our new upper bounds, indicating the optimality of our refined analysis for convergence in expectation.