Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Stochastic Optimization for Non-Smooth and Weakly Convex Problems under Heavy-Tailed Noise

Jun Hu, Chao Sun, Bo Chen, Jianzheng Wang, Zheming Wang

TL;DR

The paper tackles distributed stochastic optimization under heavy-tailed gradient noise for non-smooth, weakly convex objectives. It introduces a gradient clipping scheme within a distributed subgradient framework and leverages the Moreau envelope as a smooth surrogate to analyze convergence. The authors establish almost sure convergence of the Moreau envelope value and provide a rate bound for the gradient of the envelope under mild step-size and clipping conditions, along with a practical example in phase retrieval demonstrating robustness to heavy-tailed noise. This work advances distributed optimization by removing the bounded-variance assumption and handling non-smoothness without a central coordinator, offering theoretical guarantees and empirical validation for real-world noisy settings.

Abstract

In existing distributed stochastic optimization studies, it is usually assumed that the gradient noise has a bounded variance. However, recent research shows that the heavy-tailed noise, which allows an unbounded variance, is closer to practical scenarios in many tasks. Under heavy-tailed noise, traditional optimization methods, such as stochastic gradient descent, may have poor performance and even diverge. Thus, it is of great importance to study distributed stochastic optimization algorithms applicable to the heavy-tailed noise scenario. However, most of the existing distributed algorithms under heavy-tailed noise are developed for convex and smooth problems, which limits their applications. This paper proposes a clipping-based distributed stochastic algorithm under heavy-tailed noise that is suitable for non-smooth and weakly convex problems. The convergence of the proposed algorithm is proven, and the conditions on the parameters are given. A numerical experiment is conducted to demonstrate the effectiveness of the proposed algorithm.

Distributed Stochastic Optimization for Non-Smooth and Weakly Convex Problems under Heavy-Tailed Noise

TL;DR

The paper tackles distributed stochastic optimization under heavy-tailed gradient noise for non-smooth, weakly convex objectives. It introduces a gradient clipping scheme within a distributed subgradient framework and leverages the Moreau envelope as a smooth surrogate to analyze convergence. The authors establish almost sure convergence of the Moreau envelope value and provide a rate bound for the gradient of the envelope under mild step-size and clipping conditions, along with a practical example in phase retrieval demonstrating robustness to heavy-tailed noise. This work advances distributed optimization by removing the bounded-variance assumption and handling non-smoothness without a central coordinator, offering theoretical guarantees and empirical validation for real-world noisy settings.

Abstract

In existing distributed stochastic optimization studies, it is usually assumed that the gradient noise has a bounded variance. However, recent research shows that the heavy-tailed noise, which allows an unbounded variance, is closer to practical scenarios in many tasks. Under heavy-tailed noise, traditional optimization methods, such as stochastic gradient descent, may have poor performance and even diverge. Thus, it is of great importance to study distributed stochastic optimization algorithms applicable to the heavy-tailed noise scenario. However, most of the existing distributed algorithms under heavy-tailed noise are developed for convex and smooth problems, which limits their applications. This paper proposes a clipping-based distributed stochastic algorithm under heavy-tailed noise that is suitable for non-smooth and weakly convex problems. The convergence of the proposed algorithm is proven, and the conditions on the parameters are given. A numerical experiment is conducted to demonstrate the effectiveness of the proposed algorithm.
Paper Structure (19 sections, 7 theorems, 64 equations, 3 figures)

This paper contains 19 sections, 7 theorems, 64 equations, 3 figures.

Table of Contents

  1. INTRODUCTION
  2. NOTATIONS AND PRELIMINARIES
  3. Network
  4. Weak Convexity
  5. Moreau Envelope and Proximal Mapping
  6. PROBLEM FORMULATION
  7. ALGORITHM DESIGN
  8. Algorithm
  9. Some Technical Lemmas
  10. Convergence Analysis and Main Results
  11. NUMERICAL EXPERIMENT
  12. CONCLUSION
  13. APPENDIX
  14. Proof of \ref{['6']}
  15. Proof of Lemma 3
  16. ...and 4 more sections

Key Result

Lemma 1

chen2021distributed: Assume that $f(x)$ is $\rho$-weakly convex in $\mathbb{R}^n$. Then for $\forall x_1, \ldots, x_k \in \mathbb{R}^n$, it follows that where $\sum_{i=1}^{k}\omega_i=1$ and $\omega_i \ge 0 \text{ for }\forall i$.

Figures (3)

  • Figure 1: (a) Histogram of gradient noise samples for Phase-retrieval on MINIST dataset. (b) Histogram of samples from a sum of squared Lévy-$\alpha$-stable random variables. (c) Log-log plot for subgradient noise.
  • Figure 2: Phase retrieval results on a digit image from the MNIST dataset. From left to right: (a) The original image, (b) The image reconstructed using stoDPSM, (c) The image reconstructed using DPSM, and (d) The image reconstructed using the proposed algorithm. Each method uses $\alpha_k=30/NK$. Data size: $n=784,m=84,N=28.$
  • Figure 3: (a) Linear rate of stoDPSM, DPSM and the proposed algorithm. (b) Linear rate with different parameters.

Theorems & Definitions (11)

  • Lemma 1
  • Lemma 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 1
  • Theorem 2
  • ...and 1 more