Proximal Flow Inspired Multi-Step Methods

TL;DR

This work investigates a family of approximate multi-step proximal point methods, framed as implicit linear discretizations of gradient flow, and explores several optimization methods where applying an approximate multistep proximal points method results in improved convergence behavior.

Abstract

We investigate a family of approximate multi-step proximal point methods, framed as implicit linear discretizations of gradient flow. The resulting methods are multi-step proximal point methods, with similar computational cost in each update as the proximal point method. We explore several optimization methods where applying an approximate multistep proximal points method results in improved convergence behavior. We also include convergence analysis for the proposed method in several problem settings: quadratic problems, general problems that are strongly or weakly (non)convex, and accelerated results for alternating projections.

Figures (5)

• Figure 1: Multistep radius of convergence.$\lambda_{\max}(M)$ over changing values of $\beta$, for $\alpha = 1$ and varying $L = \lambda_{\max}(Q)$ and $m$ = number of inner gradient steps.
• Figure 2: Comparsion of different BDF schemes for proximal gradient with $\ell_1$ penalty. Data is constructed with singular values uniformly distributed (unif), with inverse decay ($1/r$), or with exponential decay ($\exp(-r)$) to simulate sensing under different sensing matrix rank conditions.
• Figure 3: Comparsion of different BDF schemes for proximal gradient with LSP (nonconvex) penalty. Data is constructed with singular values uniformly distributed (unif), with inverse decay ($1/r$), or with exponential decay ($\exp(-r)$) to simulate sensing under different sensing matrix rank conditions.
• Figure 4: Comparsion of different BDF schemes for alternating linear projections.$\sigma$ is the noise parameter that controls the angle between the subspaces; smaller means more ill-conditioned.
• Figure 5: Comparsion of BDF schemes for alternating minimizations for matrix factorization. Here $r$ is the matrix rank and $\alpha$ the step size of the inner loop.

Theorems & Definitions (31)

• Definition 2.1
• Definition 2.2
• Proposition 2.1
• proof
• Lemma 2.2
• proof : Proof of Lemma \ref{['lem:exist_unique']}
• Lemma 2.3
• proof
• Theorem 2.4
• proof