Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Generalized Alternating Anderson Acceleration Method

Yunhui He, Santolo Leveque

TL;DR

The paper introduces a generalized alternating Anderson acceleration framework, $aAA(m)[$s$]--FP[$t$], which interleaves $t$ fixed-point steps with $s$ steps of AA($m$) to accelerate linear and nonlinear fixed-point iterations. It establishes a theoretical connection between $aAA( ext{infty})$[1]--FP[$t$] and GMRES for linear problems, and provides sufficient convergence conditions when the fixed-point matrix is diagonalizable and noncontractive. The authors demonstrate broad applicability by accelerating Jacobi, Gauss–Seidel, Picard, gradient descent, and ADMM across PDEs and nonlinear optimization, with numerical results showing substantial improvements in iteration counts and CPU time under carefully chosen $(m,s,t)$. These findings offer a flexible, scalable approach to speeding up slow fixed-point solvers in challenging linear and nonlinear contexts, while highlighting practical considerations like parameter tuning and potential rank-deficiency in large LS problems.

Abstract

In this work, we propose a generalized alternating Anderson acceleration method, a periodic scheme composed of $t$ fixed-point iteration steps, interleaved with $s$ steps of Anderson acceleration with window size $m$, to solve linear and nonlinear problems. This allows flexibility to use different combinations of fixed-point iteration and Anderson iteration. We present a convergence analysis of the proposed scheme for accelerating the Richardson iteration in the linear case, with a focus on specific parameter choices of interest. Specifically, we prove convergence of the proposed method under contractive fixed-point iteration and provide a sufficient condition for convergence when the Richardson iteration matrix is diagonalizable and noncontractive. To demonstrate the broader applicability of our proposed method, we use it to accelerate Jacobi iteration, Picard iteration, gradient descent, and the alternating direction method of multipliers in solving partial differential equations and nonlinear, nonsmooth optimization problems. The numerical results illustrate that the proposed scheme is more efficient than the existing windowed Anderson acceleration and alternating Anderson ($s=1$) in terms of iteration number and CPU time for careful choice of parameters $m, s, t$.

A Generalized Alternating Anderson Acceleration Method

TL;DR

The paper introduces a generalized alternating Anderson acceleration framework, sttsmaAA( ext{infty})t(m,s,t)$. These findings offer a flexible, scalable approach to speeding up slow fixed-point solvers in challenging linear and nonlinear contexts, while highlighting practical considerations like parameter tuning and potential rank-deficiency in large LS problems.

Abstract

In this work, we propose a generalized alternating Anderson acceleration method, a periodic scheme composed of fixed-point iteration steps, interleaved with steps of Anderson acceleration with window size , to solve linear and nonlinear problems. This allows flexibility to use different combinations of fixed-point iteration and Anderson iteration. We present a convergence analysis of the proposed scheme for accelerating the Richardson iteration in the linear case, with a focus on specific parameter choices of interest. Specifically, we prove convergence of the proposed method under contractive fixed-point iteration and provide a sufficient condition for convergence when the Richardson iteration matrix is diagonalizable and noncontractive. To demonstrate the broader applicability of our proposed method, we use it to accelerate Jacobi iteration, Picard iteration, gradient descent, and the alternating direction method of multipliers in solving partial differential equations and nonlinear, nonsmooth optimization problems. The numerical results illustrate that the proposed scheme is more efficient than the existing windowed Anderson acceleration and alternating Anderson () in terms of iteration number and CPU time for careful choice of parameters .

Paper Structure

This paper contains 16 sections, 7 theorems, 81 equations, 13 figures, 11 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Generalized alternating Anderson acceleration
  3. Convergence analysis for the linear case
  4. Numerical results
  5. Validation
  6. The nonsymmetric case
  7. The symmetric case
  8. Application of aAA--FP to nonsymmetric cases
  9. Application of aAA--FP to the incompressible Navier--Stokes equations
  10. Application of aAA--FP to nonlinear optimization
  11. Total variation
  12. Lasso problem
  13. Nonnegative least squares problem
  14. Regularized logistic regression
  15. Study the parameters choice
  16. ...and 1 more sections

Key Result

Theorem 1

Consider the aAA($\infty$)[$1$]--FP[$t$] method described in Algorithm saAAm_FPn, applied to solve eq:Ax=b using the fixed-point iteration eq:FP-linear. Denote its iterates as $\mathbf{x}_k$. Let $p=t+1$. Assume that there exists a $k_0$ such that the norms of the residuals of GMRES are strictly dec where $\mathbf{x}_{jp-1}^G$ is the $(jp-1)$th iterate of GMRES.

Figures (13)

  • Figure 1: Diagram loop of aAA(3)[1]--FP[3]
  • Figure 2: Diagram loop of aAA(2)[2]--FP[1]
  • Figure 3: Diagram loop of aAA(3)[3]--FP[5]
  • Figure 4: Diagram loop of aAA($\infty$)[1]--FP[2]
  • Figure 5: Plots of the norms of $\mathbf{M}\mathbf{r}_k^{G}$ and $\mathbf{r}_k$ for the nonsymmetric case for aAA($\infty$)[1]--FP[3]. Left: $n=32$. Right: $n=26$.
  • ...and 8 more figures

Theorems & Definitions (14)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Lemma 1
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • ...and 4 more