Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constrained Optimization with Compressed Gradients: A Dynamical Systems Perspective

Zhaoyue Xia, Jun Du, Chunxiao Jiang, H. Vincent Poor, Yong Ren

TL;DR

This work establishes a tight connection between a continuous-time nonsmooth dynamical system called a perturbed sweeping process (PSP) and a projected scheme with compressed gradients and proposes a provably convergent distributed compressed gradient descent algorithm for distributed nonconvex optimization.

Abstract

Gradient compression is of growing interests for solving constrained optimization problems including compressed sensing, noisy recovery and matrix completion under limited communication resources and storage costs. Convergence analysis of these methods from the dynamical systems viewpoint has attracted considerable attention because it provides a geometric demonstration towards the shadowing trajectory of a numerical scheme. In this work, we establish a tight connection between a continuous-time nonsmooth dynamical system called a perturbed sweeping process (PSP) and a projected scheme with compressed gradients. Theoretical results are obtained by analyzing the asymptotic pseudo trajectory of a PSP. We show that under mild assumptions a projected scheme converges to an internally chain transitive invariant set of the corresponding PSP. Furthermore, given the existence of a Lyapunov function $V$ with respect to a set $Λ$, convergence to $Λ$ can be established if $V(Λ)$ has an empty interior. Based on these theoretical results, we are able to provide a useful framework for convergence analysis of projected methods with compressed gradients. Moreover, we propose a provably convergent distributed compressed gradient descent algorithm for distributed nonconvex optimization. Finally, numerical simulations are conducted to confirm the validity of theoretical analysis and the effectiveness of the proposed algorithm.

Constrained Optimization with Compressed Gradients: A Dynamical Systems Perspective

TL;DR

This work establishes a tight connection between a continuous-time nonsmooth dynamical system called a perturbed sweeping process (PSP) and a projected scheme with compressed gradients and proposes a provably convergent distributed compressed gradient descent algorithm for distributed nonconvex optimization.

Abstract

Gradient compression is of growing interests for solving constrained optimization problems including compressed sensing, noisy recovery and matrix completion under limited communication resources and storage costs. Convergence analysis of these methods from the dynamical systems viewpoint has attracted considerable attention because it provides a geometric demonstration towards the shadowing trajectory of a numerical scheme. In this work, we establish a tight connection between a continuous-time nonsmooth dynamical system called a perturbed sweeping process (PSP) and a projected scheme with compressed gradients. Theoretical results are obtained by analyzing the asymptotic pseudo trajectory of a PSP. We show that under mild assumptions a projected scheme converges to an internally chain transitive invariant set of the corresponding PSP. Furthermore, given the existence of a Lyapunov function with respect to a set , convergence to can be established if has an empty interior. Based on these theoretical results, we are able to provide a useful framework for convergence analysis of projected methods with compressed gradients. Moreover, we propose a provably convergent distributed compressed gradient descent algorithm for distributed nonconvex optimization. Finally, numerical simulations are conducted to confirm the validity of theoretical analysis and the effectiveness of the proposed algorithm.
Paper Structure (13 sections, 16 theorems, 108 equations, 7 figures, 1 algorithm)

This paper contains 13 sections, 16 theorems, 108 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Basic Concepts and Notation
  3. From Continuous Dynamics to Cascades
  4. Results on Perturbed Sweeping Processes
  5. Explicit Euler Scheme with Decaying Step Sizes
  6. Application to Constrained Optimization
  7. Schemes with Exact Inputs
  8. Schemes with Compressors
  9. A Distributed Scheme with Compressors
  10. Numerical Simulation
  11. Centralized Methods with Exact Gradients
  12. The Distributed Scheme with Compressed Gradients
  13. Conclusion

Key Result

Lemma 1

Let $\mathcal{H}$ be a Hilbert space. Let functions $V,W: \mathbb{R} \times \mathcal{H} \to \mathbb{R}$ be lower semi-continuous, with $W \ge 0$. $(V,W)$ is a time-dependent Lyapunov pair if and only if for all $t \ge 0$, $x \in \mathcal{H}$ and $\xi \in \partial_F V(t,x)$, we have

Figures (7)

  • Figure 1: The conceptual framework to design a convergent projected method.
  • Figure 2: Results of constrained convex optimization.
  • Figure 3: Six-hump camel back function.
  • Figure 4: Results of optimizing the six-hump camel back function within the area $[-1,1]^2$.
  • Figure 5: Results of optimizing the six-hump camel back function with methods incorporating random perturbations.
  • ...and 2 more figures

Theorems & Definitions (33)

  • Definition 1: A variant of Definition 1 in 2012LyapunovPair
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • proof
  • Lemma 5
  • proof
  • ...and 23 more