Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping
Zijian Liu, Zhengyuan Zhou
TL;DR
The paper addresses nonconvex stochastic optimization under heavy-tailed noise, where the stochastic gradient has only a finite $\mathfrak{p}$-th moment with $\mathfrak{p}\in(1,2]$. It introduces Batched NSGDM with gradient normalization to achieve clipping-free convergence, establishing the optimal rate $O\left(T^{\frac{1-\mathfrak{p}}{3\mathfrak{p}-2}}\right)$ when $\mathfrak{p}$ is known and a robust rate $O\left(T^{\frac{1-\mathfrak{p}}{2\mathfrak{p}}}\right)$ when it is unknown. The analysis hinges on a generalized heavy-tailed noise model and a new vector-valued martingale inequality (main-core), which shows gradient normalization can substitute clipping by reducing the error term’s effective order and enabling adaptivity to $\mathfrak{p}$. The work extends convergence guarantees to generalized smoothness and provides a refined lower bound aligning problem-dependent factors, with practical implications for training under heavy-tailed noise without gradient clipping.
Abstract
Recently, the study of heavy-tailed noises in first-order nonconvex stochastic optimization has gotten a lot of attention since it was recognized as a more realistic condition as suggested by many empirical observations. Specifically, the stochastic noise (the difference between the stochastic and true gradient) is considered to have only a finite $\mathfrak{p}$-th moment where $\mathfrak{p}\in\left(1,2\right]$ instead of assuming it always satisfies the classical finite variance assumption. To deal with this more challenging setting, people have proposed different algorithms and proved them to converge at an optimal $\mathcal{O}(T^{\frac{1-\mathfrak{p}}{3\mathfrak{p}-2}})$ rate for smooth objectives after $T$ iterations. Notably, all these new-designed algorithms are based on the same technique - gradient clipping. Naturally, one may want to know whether the clipping method is a necessary ingredient and the only way to guarantee convergence under heavy-tailed noises. In this work, by revisiting the existing Batched Normalized Stochastic Gradient Descent with Momentum (Batched NSGDM) algorithm, we provide the first convergence result under heavy-tailed noises but without gradient clipping. Concretely, we prove that Batched NSGDM can achieve the optimal $\mathcal{O}(T^{\frac{1-\mathfrak{p}}{3\mathfrak{p}-2}})$ rate even under the relaxed smooth condition. More interestingly, we also establish the first $\mathcal{O}(T^{\frac{1-\mathfrak{p}}{2\mathfrak{p}}})$ convergence rate in the case where the tail index $\mathfrak{p}$ is unknown in advance, which is arguably the common scenario in practice.