Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated forward-backward and Douglas-Rachford splitting dynamics

Ibrahim K. Ozaslan, Mihailo R. Jovanović

Abstract

We examine convergence properties of continuous-time variants of accelerated Forward-Backward (FB) and Douglas-Rachford (DR) splitting algorithms for nonsmooth composite optimization problems. When the objective function is given by the sum of a quadratic and a nonsmooth term, we establish accelerated sublinear and exponential convergence rates for convex and strongly convex problems, respectively. Moreover, for FB splitting dynamics, we demonstrate that accelerated exponential convergence rate carries over to general strongly convex problems. In our Lyapunov-based analysis we exploit the variable-metric gradient interpretations of FB and DR splittings to obtain smooth Lyapunov functions that allow us to establish accelerated convergence rates. We provide computational experiments to demonstrate the merits and the effectiveness of our analysis.

Accelerated forward-backward and Douglas-Rachford splitting dynamics

Abstract

We examine convergence properties of continuous-time variants of accelerated Forward-Backward (FB) and Douglas-Rachford (DR) splitting algorithms for nonsmooth composite optimization problems. When the objective function is given by the sum of a quadratic and a nonsmooth term, we establish accelerated sublinear and exponential convergence rates for convex and strongly convex problems, respectively. Moreover, for FB splitting dynamics, we demonstrate that accelerated exponential convergence rate carries over to general strongly convex problems. In our Lyapunov-based analysis we exploit the variable-metric gradient interpretations of FB and DR splittings to obtain smooth Lyapunov functions that allow us to establish accelerated convergence rates. We provide computational experiments to demonstrate the merits and the effectiveness of our analysis.
Paper Structure (19 sections, 6 theorems, 83 equations, 2 figures)

This paper contains 19 sections, 6 theorems, 83 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem formulation and preliminaries
  3. Proximal operator and Moreau envelope
  4. The FB and DR envelopes
  5. The FB and DR splittings
  6. Main results
  7. Convex quadratic $f$
  8. Strongly convex $f$
  9. Proof of main results
  10. Theorems \ref{['theorem.quad_cvx']} and \ref{['theorem.quad_str']}: convex quadratic $f$
  11. Theorem 1: convex quadratic $f$
  12. Theorem 2: strongly convex quadratic $f$
  13. Theorem 3: strongly convex $f$
  14. Computational experiments
  15. Example 1: $\ell_1$-regularized least squares
  16. ...and 4 more sections

Key Result

Lemma 1

Let Assumption ass.1 hold with $m>0$ and let $\mu = 1/(2L)$. Then, the proximal gradient flow eq.fb_dyn and DR splitting dynamics eq.dr_dyn are globally exponentially stable with rate $\rho=\alpha m$.

Figures (2)

  • Figure 1: Plot of the function $h$ that demonstrates $h < 0$. Thus, $w h(w)\leq0$ for $w\in[0,1]$ and condition (iii) is satisfied.
  • Figure 2: (a) Sublinear convergence of accelerated dynamics \ref{['eq.acc_fb']} and \ref{['eq.acc_dr']} with the time-varying parameters given in Theorem \ref{['theorem.quad_cvx']} for $\ell_1$-regularized least squares problem \ref{['eq.ls-ell1']}. (b) Linear convergence of accelerated dynamics \ref{['eq.acc_fb']} and \ref{['eq.acc_dr']} with the constant parameters given in Theorem \ref{['theorem.quad_str']} for box-constrained quadratic program \ref{['eq.qp-box']}. (c) Linear convergence of accelerated dynamics \ref{['eq.acc_fb']} eith the constant parameters given in Theorem \ref{['theorem.gen_str']} and $\mu = 1/(L\sqrt[4]{\kappa}) \approx 1/(26L)$ for the $\ell_1$-regularized logistic regression problem \ref{['eq.l1-logistic']} with $\varrho = 0.1$, and condition number $\kappa\approx5\cdot10^5$.

Theorems & Definitions (7)

  • Lemma 1: Thms. 3 and 7, mogjovAUT21
  • Theorem 1
  • Lemma 2: Thm. 2.3 patstebem14 and Proposition 4.6 gisfae18
  • Theorem 2
  • Theorem 3
  • Remark 1
  • Lemma 3