Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Decentralized Nonconvex Algorithm under Kurdyka-Łojasiewicz Property

Hao Wu, Liping Wang, Hongchao Zhang

TL;DR

This paper proposes a unified decentralized nonconvex algorithmic framework that subsumes existing state-of-the-art gradient tracking algorithms and particularly several quasi-Newton algorithms and proposes some quasi-Newton variants that fit into this framework.

Abstract

In this paper, we study the decentralized optimization problem of minimizing a finite sum of continuously differentiable and possibly nonconvex functions over a fixed-connected undirected network. We propose a unified decentralized nonconvex algorithmic framework that subsumes existing state-of-the-art gradient tracking algorithms and particularly several quasi-Newton algorithms. We present a general analytical framework for the convergence of our unified algorithm under both nonconvex and the Kurdyka-Łojasiewicz condition settings. We also propose some quasi-Newton variants that fit into our framework, where economical implementation strategies are derived for ensuring bounded eigenvalues of Hessian inverse approximations. Our numerical results show that these newly developed algorithms are very efficient compared with other state-of-the-art algorithms for solving decentralized nonconvex smooth optimization.

A Unified Decentralized Nonconvex Algorithm under Kurdyka-Łojasiewicz Property

TL;DR

This paper proposes a unified decentralized nonconvex algorithmic framework that subsumes existing state-of-the-art gradient tracking algorithms and particularly several quasi-Newton algorithms and proposes some quasi-Newton variants that fit into this framework.

Abstract

In this paper, we study the decentralized optimization problem of minimizing a finite sum of continuously differentiable and possibly nonconvex functions over a fixed-connected undirected network. We propose a unified decentralized nonconvex algorithmic framework that subsumes existing state-of-the-art gradient tracking algorithms and particularly several quasi-Newton algorithms. We present a general analytical framework for the convergence of our unified algorithm under both nonconvex and the Kurdyka-Łojasiewicz condition settings. We also propose some quasi-Newton variants that fit into our framework, where economical implementation strategies are derived for ensuring bounded eigenvalues of Hessian inverse approximations. Our numerical results show that these newly developed algorithms are very efficient compared with other state-of-the-art algorithms for solving decentralized nonconvex smooth optimization.

Paper Structure

This paper contains 15 sections, 16 theorems, 118 equations, 6 figures, 4 tables, 4 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Notation
  3. ALGORITHM DEVELOPMENT, CONVERGENCE ANALYSIS, AND PRACTICAL IMPLEMENTATION
  4. Unified Nonconvex Algorithm Framework
  5. Global convergence
  6. Convergence under KŁ property
  7. Methods of constructing Hessian inverse approximations
  8. Memoryless quasi-Newton updating schemes
  9. Memoryless SR1
  10. Memoryless BFGS
  11. Corrected quasi-Newton updating schemes
  12. NUMERICAL EXPERIMENTS
  13. CONCLUSIONS
  14. Analytical tools
  15. Algorithmic procedure for UDNAs

Key Result

Lemma 2.2

For $\tilde{{\bf{W}}}$ defined in Definition mix and ${\bf{W}} :=\tilde{{\bf{W}}}\otimes{\bf{I}}_p$, we have

Figures (6)

  • Figure 1: Optimality error of UDNAs for minimizing the nonconvex logistic regression problem \ref{['noncovex_logistic_problem']} on different datasets w.r.t. communication volume.
  • Figure 2: Comparisons with gradient-based algorithms for minimizing the nonconvex logistic regression problem \ref{['noncovex_logistic_problem']} on different datasets w.r.t. communication volume.
  • Figure 3: Comparisons with quasi-Newton algorithms for minimizing the nonconvex logistic regression problem \ref{['noncovex_logistic_problem']} on different datasets w.r.t. number of iteration.
  • Figure 4: Comparisons with quasi-Newton algorithms for minimizing the nonconvex logistic regression problem \ref{['noncovex_logistic_problem']} on different datasets w.r.t. communication volume.
  • Figure 5: Comparisons with quasi-Newton algorithms for minimizing the nonconvex logistic regression problem \ref{['noncovex_logistic_problem']} on colon-cancer datasets.
  • ...and 1 more figures

Theorems & Definitions (32)

  • Definition 2.1
  • Lemma 2.2
  • proof
  • Remark 2.3
  • Lemma 2.4
  • proof
  • Theorem 2.5
  • proof
  • Corollary 2.6
  • proof
  • ...and 22 more