Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noise
Menglian Wang, Zhuanghua Liu, Luo Luo
TL;DR
The paper tackles decentralized stochastic nonconvex optimization under heavy-tailed gradient noise on directed graphs. It introduces DNSGD-PD, a method that combines gradient normalization with Pull-Diag gradient tracking and multi-consensus gossip to handle directed communication and $p$-BCM noise ($p\in(1,2]$). The authors prove that the method achieves an $\epsilon$-stationary point with near-optimal sample and communication complexities, $\mathcal{O}(L\sigma^{\frac{p}{p-1}}\epsilon^{-\frac{3p-2}{p-1}}\Delta)$ and $\tilde{\mathcal{O}}(L(1-\beta)^{-1}\epsilon^{-2}\Delta)$ respectively, and extend the results to undirected networks with near-tight bounds. They also provide lower-bound arguments to establish optimality and demonstrate practical benefits via experiments on Transformer-XL language modeling. Overall, the work advances decentralized optimization by handling heavy-tailed noise and directed topologies with provably efficient convergence.
Abstract
This paper studies decentralized stochastic nonconvex optimization problem over row-stochastic networks. We consider the heavy-tailed gradient noise which is empirically observed in many popular real-world applications. Specifically, we propose a decentralized normalized stochastic gradient descent with Pull-Diag gradient tracking, which achieves approximate stationary points with the optimal sample complexity and the near-optimal communication complexity. We further follow our framework to study the setting of undirected networks, also achieving the nearly tight upper complexity bounds. Moreover, we conduct empirical studies to show the practical superiority of the proposed methods.