Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reduced order method based Anderson-type acceleration method for nonlinear least square problems and large scale ill-posed problems

Kazufumi Ito, Tiancheng Xue

TL;DR

The paper develops a Reduced Order Method (ROM)–based acceleration framework and couples it with Anderson-type updates to enhance convergence of iterative schemes for nonlinear least squares and ill-posed problems. By exploiting the subspace spanned by historical iterates, it minimizes equation error or residual over that manifold, yielding monotone convergence and regularization effects. It clarifies the relationship between ROM and Anderson in linear settings, introduces variants such as Nested ROM and Sampled ROM, and extends to practical applications including matrix Riccati equations and saddle-point problems with numerical continuation to handle non-contractive regimes. Together, these contributions deliver a flexible, history-based acceleration strategy applicable to a broad class of fixed-point and optimization problems, improving robustness and performance on large-scale ill-posed systems.

Abstract

In this paper, we propose an acceleration framework for a class of iterative methods using the Reduced Order Method (ROM). Assuming that the underlying iterative scheme generates a rich basis for the solution space, we construct the next iterate by minimizing the equation error over the linear manifold spanned by this basis. The resulting optimal linear combination yields a more accurate approximation of the solution and significantly enhances convergence. In essence, the method can be seen as a history-based acceleration technique, akin to a delayed or memory-enhanced iterative scheme. This approach effectively remedies semi-ill-posed problems, enabling convergence where standard methods may fail, and also acts as a stabilizing and regularizing mechanism for the original iteration.

Reduced order method based Anderson-type acceleration method for nonlinear least square problems and large scale ill-posed problems

TL;DR

The paper develops a Reduced Order Method (ROM)–based acceleration framework and couples it with Anderson-type updates to enhance convergence of iterative schemes for nonlinear least squares and ill-posed problems. By exploiting the subspace spanned by historical iterates, it minimizes equation error or residual over that manifold, yielding monotone convergence and regularization effects. It clarifies the relationship between ROM and Anderson in linear settings, introduces variants such as Nested ROM and Sampled ROM, and extends to practical applications including matrix Riccati equations and saddle-point problems with numerical continuation to handle non-contractive regimes. Together, these contributions deliver a flexible, history-based acceleration strategy applicable to a broad class of fixed-point and optimization problems, improving robustness and performance on large-scale ill-posed systems.

Abstract

In this paper, we propose an acceleration framework for a class of iterative methods using the Reduced Order Method (ROM). Assuming that the underlying iterative scheme generates a rich basis for the solution space, we construct the next iterate by minimizing the equation error over the linear manifold spanned by this basis. The resulting optimal linear combination yields a more accurate approximation of the solution and significantly enhances convergence. In essence, the method can be seen as a history-based acceleration technique, akin to a delayed or memory-enhanced iterative scheme. This approach effectively remedies semi-ill-posed problems, enabling convergence where standard methods may fail, and also acts as a stabilizing and regularizing mechanism for the original iteration.
Paper Structure (24 sections, 10 theorems, 119 equations, 4 figures, 4 tables)

This paper contains 24 sections, 10 theorems, 119 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Various interactive methods
  3. Fixed point problem: $F(x)=0$
  4. Quasi Newton
  5. Conjugate gradient
  6. Nonlinear conjugate gradient
  7. Solution extrapolation
  8. ROM, Anderson acceleration, and Evolutional Fixed point
  9. ROM in general
  10. Anderson type acceleration
  11. Comparison of Anderson and ROM
  12. Nested ROM
  13. Sampled ROM
  14. ROM extrapolation
  15. Variable step size fixed point
  16. ...and 9 more sections

Key Result

Theorem 3.1

Given basis vectors $(x_1, \cdots, x_m) \in X \times \cdots \times X$, with $e_i = F(x_i)$ for $i \in \{ 1,\cdots,m \}$, consider the problem: With $B_{ij} = ( e_i , e_j )$, the optimal solution $c_1, \cdots, c_n$ is obtained by solving the linear equation

Figures (4)

  • Figure 1: Saddle Point Problem 1 (One direction update: $r = b-Ax$): comparison of standard case, accelerated case, and sampled case.
  • Figure 2: Saddle Point Problem 1 (Two directions' update: $r = b-Ax$, $Ar$): comparison of standard case, accelerated case, and sampled case.
  • Figure 3: Saddle Point Problem 2 (One direction update $r=b-Ax$): comparison of standard case, accelerated case, and sampled case.
  • Figure 4: Saddle Point Problem 2 (Two directions' update $r=b-Ax$, $\bar{A}r$): comparison of standard case, accelerated case, and sampled case.

Theorems & Definitions (30)

  • Remark 2.1
  • Theorem 3.1
  • Proof 1
  • Remark 3.2
  • Definition 4.1
  • Definition 4.2
  • Proposition 4.3
  • Proof 2
  • Theorem 4.4
  • Proof 3
  • ...and 20 more