Convergence Analysis for Nonlinear GMRES

TL;DR

This work analyzes the convergence of nonlinear GMRES (NGMRES) when used to accelerate fixed-point iterations for nonlinear systems, focusing on the residuals $r_k=r(x_k)$ with $x_{k+1}=q(x_k)=x_k-g(x_k)$. It proves r-linear convergence of NGMRES($m$) under contraction-type assumptions and a bounded-coefficient condition, and establishes q-linear convergence for NGMRES(0) with a rate $\\eta=\frac{\rho(1+\rho)}{1-\\rho}$ given $0<\rho<\\sqrt{2}-1$. Numerical experiments on problems including a two-variable nonlinear system with multiple fixed points and a high-dimensional trig example validate the theory and demonstrate substantial acceleration over the base fixed-point iterations. The results underscore both the practical potential of windowed NGMRES and the need for further analysis for noncontractive operators and sharpened rate estimates.

Abstract

In this work, we revisit nonlinear generalized minimal residual method (NGMRES) applied to nonlinear problems. NGMRES is used to accelerate the convergence of fixed-point iterations, which can substantially improve the performance of the underlying fixed-point iterations. We consider NGMRES with a finite window size $m$, denoted as NGMRES($m$). However, there is no convergence analysis for NGMRES($m$) applied to nonlinear systems. We prove that for general $m>0$, the residuals of NGMRES($m$) converge r-linearly under some conditions. For $m=0$, we prove that the residuals of NGMRES(0) converge q-linearly.

Figures (5)

• Figure 1: Example \ref{['ex:2nonlinear-contr']}. Left panel: q-convergence factor $\|r_k\|/\|r_{k-1}\|$ as a function of the iteration index $k$ with 1000 random initial guesses $x_0=\frac{2y}{5\|y\|}$, where $y \in(-1,1)^2$. Right panel: convergence history of NGMRES(0) and FP with initial guess $x_0=[-0.25, 0.25]^T$ for $c_1=\frac{4}{5}, c_2=\frac{2}{3}$.
• Figure 2: Example \ref{['ex:2nonlinear-contr']}. Left panel: convergence history of NGMRES(0) and FP with initial guess $x_0=[-0.25, 0.25]^T$ for $c_1=c_2=1$. Right panel: convergence history of NGMRES(0) and NGMRES(1) with initial guess $x_0=[-0.25, 0.25]^T$ for $c_1=1, c_2=2$.
• Figure 3: Example \ref{['ex:trig']}. Left panel: convergence history $\|r_k\|$ as a function of iteration index $k$ with 1000 random initial guesses. Right panel: the corresponding root-convergence factor $\|r_k\|^{1/k}$ as a function of the iteration index $k$ with 1000 random initial guesses.
• Figure 4: Final residual norm versus initial distance $d=\|x_0-x^*\|$ for FP and NGMRES($m$). Left panel is for Example \ref{['ex:2nonlinear-contr']} with $c_1=c_2=1$ and $x^*=[0, 0]^T$. Right panel is for Example \ref{['ex:trig']}.
• Figure 5: Example \ref{['ex:tworepeatroot']}: Convergence history. Left panel is for $c_1=3/4$. Right panel is for $c_2=1/4$.

Theorems & Definitions (11)

• Definition 2.1
• Definition 2.2
• Lemma 1
• proof
• Theorem 1
• proof
• Theorem 2
• proof
• Example 3.1
• Example 3.2: Trigonometric functions, $s=100$