Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Nonsmooth Nonconvex Optimization with Client Sampling

Xinyan Chen, Weiguo Gao, Luo Luo

TL;DR

An efficient stochastic first-order method with client sampling achieves the optimal sample complexity and the sharper communication complexity than existing methods and extends the ideas to zeroth-order optimization.

Abstract

This paper considers decentralized nonsmooth nonconvex optimization problem with Lipschitz continuous local functions. We propose an efficient stochastic first-order method with client sampling, achieving the $(δ,ε)$-Goldstein stationary point with the overall sample complexity of ${\mathcal O}(δ^{-1}ε^{-3})$, the computation rounds of ${\mathcal O}(δ^{-1}ε^{-3})$, and the communication rounds of ${\tilde{\mathcal O}}(γ^{-1/2}δ^{-1}ε^{-3})$, where $γ$ is the spectral gap of the mixing matrix for the network. Our results achieve the optimal sample complexity and the sharper communication complexity than existing methods. We also extend our ideas to zeroth-order optimization. Moreover, the numerical experiments show the empirical advantage of our methods.

Decentralized Nonsmooth Nonconvex Optimization with Client Sampling

TL;DR

An efficient stochastic first-order method with client sampling achieves the optimal sample complexity and the sharper communication complexity than existing methods and extends the ideas to zeroth-order optimization.

Abstract

This paper considers decentralized nonsmooth nonconvex optimization problem with Lipschitz continuous local functions. We propose an efficient stochastic first-order method with client sampling, achieving the -Goldstein stationary point with the overall sample complexity of , the computation rounds of , and the communication rounds of , where is the spectral gap of the mixing matrix for the network. Our results achieve the optimal sample complexity and the sharper communication complexity than existing methods. We also extend our ideas to zeroth-order optimization. Moreover, the numerical experiments show the empirical advantage of our methods.
Paper Structure (29 sections, 16 theorems, 109 equations, 10 figures, 1 table, 4 algorithms)

This paper contains 29 sections, 16 theorems, 109 equations, 10 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation and Assumptions
  4. Goldstein Stationary Points
  5. Randomized Smoothing
  6. The Algorithm and Main Results
  7. The Complexity Analysis
  8. Numerical Experiments
  9. Conclusion
  10. Upper Bounds for Consensus Errors
  11. Proof of Lemma \ref{['lemma:OL communicate']}
  12. Proof of Lemma \ref{['lem:dec_bound']}
  13. Induction Base:
  14. Induction Step:
  15. Proofs for the Smooth Case
  16. ...and 14 more sections

Key Result

Proposition 1

Suppose the function $f : \mathbb{R}^d \rightarrow \mathbb{R}$ is $L$-Lipschitz, then its smooth surrogate $f_\delta$ holds: (a) $|f_\delta(\cdot) - f(\cdot)| \le \delta L$; (b) $f_\delta(\cdot)$ is $L$-Lipschitz; (c) $f_\delta(\cdot)$ is differentiable with $c_0\sqrt{d}L \delta^{-1}$-Lipschitz grad

Figures (10)

  • Figure 1: Sample complexity of first-order methods on all tasks.
  • Figure 2: Computation Rounds of first-order methods on all tasks.
  • Figure 3: Communication Rounds of first-order methods on all tasks.
  • Figure 4: Computation Rounds of zeroth-order methods on all tasks.
  • Figure 5: Computation Rounds of zeroth-order methods on all tasks.
  • ...and 5 more figures

Theorems & Definitions (30)

  • Definition 1: clarke2008nonsmooth
  • Definition 2: goldstein1977optimization
  • Definition 3: zhang2020complexity
  • Definition 4: yousefian2012stochastic
  • Proposition 1: lin2022gradient, kornowski2023algorithm
  • Lemma 1: kornowski2023algorithm
  • Theorem 1
  • Corollary 2
  • Theorem 3
  • Corollary 4
  • ...and 20 more