Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification

Kexin Li, Luwei Bai, Xiao Wang, Hao Wang

TL;DR

This paper investigates a class of nonsmooth optimization algorithms characterized by the active manifold identification property and establishes a local R-linear convergence rate for the Anderson-accelerated algorithm.

Abstract

Anderson acceleration is an effective technique for enhancing the efficiency of fixed-point iterations; however, analyzing its convergence in nonsmooth settings presents significant challenges. In this paper, we investigate a class of nonsmooth optimization algorithms characterized by the active manifold identification property. This class includes a diverse array of methods such as the proximal point method, proximal gradient method, proximal linear method, proximal coordinate descent method, Douglas-Rachford splitting (or the alternating direction method of multipliers), and the iteratively reweighted $\ell_1$ method, among others. Under the assumption that the optimization problem possesses an active manifold at a stationary point, we establish a local R-linear convergence rate for the Anderson-accelerated algorithm. Our extensive numerical experiments further highlight the robust performance of the proposed Anderson-accelerated methods.

Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification

TL;DR

This paper investigates a class of nonsmooth optimization algorithms characterized by the active manifold identification property and establishes a local R-linear convergence rate for the Anderson-accelerated algorithm.

Abstract

Anderson acceleration is an effective technique for enhancing the efficiency of fixed-point iterations; however, analyzing its convergence in nonsmooth settings presents significant challenges. In this paper, we investigate a class of nonsmooth optimization algorithms characterized by the active manifold identification property. This class includes a diverse array of methods such as the proximal point method, proximal gradient method, proximal linear method, proximal coordinate descent method, Douglas-Rachford splitting (or the alternating direction method of multipliers), and the iteratively reweighted method, among others. Under the assumption that the optimization problem possesses an active manifold at a stationary point, we establish a local R-linear convergence rate for the Anderson-accelerated algorithm. Our extensive numerical experiments further highlight the robust performance of the proposed Anderson-accelerated methods.

Paper Structure

This paper contains 14 sections, 9 theorems, 53 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation and preliminaries
  3. Anderson acceleration
  4. Subdifferentials
  5. Weak convexity and Moreau envelope
  6. Manifold optimization
  7. Anderson-accelerated proximal algorithms
  8. Anderson-accelerated proximal coordinate descent algorithm
  9. Anderson-accelerated DRS/ADMM
  10. Anderson-accelerated iteratively reweighted l1 algorithm
  11. Numerical experiments
  12. PGA for sparse signal recovery
  13. PCD, DRS, and IRL1 for classification problems
  14. Conclusions

Key Result

Theorem 1

Under Assumption Assumption AA, suppose that $H$ is Lipschitz continuously differentiable in a neighborhood $\mathcal{B}_{\bm{x}^*}(\hat{\rho})$ of $\bm{x}^*$ for some $0 < \hat{\rho} \le \rho$ and $\bm{x}^0 \in \mathcal{B}_{\bm{x}^*}(\hat{\rho})$. Then, when $\bm{x}^0$ is sufficiently close to $\bm

Figures (6)

  • Figure 1: Flow of theoretical analysis to Anderson-accelerated nonsmooth optimization algorithms for \ref{['Nonsmooth optimization problem']} with $\bm x^*$ being a Clarke critical point.
  • Figure 2: Landscapes of commonly used 2-dimensional nonsmooth functions $f(x,y)$.
  • Figure 3: Comparison of $\|\bm{r}^k\|$ for ISTA, FISTA, AAISTA under different $(M,N)$.
  • Figure 4: Comparison of $\|\bm{r}^k\|$ for PCD and AAPCD(m).
  • Figure 5: Comparison of $\|\bm{r}^k\|$ for DRS and AADRS(m).
  • ...and 1 more figures

Theorems & Definitions (19)

  • Theorem 1
  • Lemma 2.1
  • Definition 2.1: Smooth manifold
  • Definition 2.2: Partly smooth function and active manifold
  • Definition 2.3: Active manifold identification
  • Lemma 2.2: Sensitivity analysis
  • Theorem 2
  • proof
  • Definition 3.1: Composite active manifold
  • Theorem 3
  • ...and 9 more