Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Combination of Preconditioned Gradient Descent Methods and Vector Extrapolation Techniques for Nonlinear Least-Squares Problems

Abdellatif Mouhssine

TL;DR

The paper addresses the slow convergence of nonlinear least-squares solvers by coupling vector extrapolation techniques (RRE, MPE, VEA) with preconditioned gradient descent (PGD, SGD). It introduces restarted extrapolation frameworks and applies them to solve large-scale problems, notably the Bratu equation and an extremely sparse nonlinear model, benchmarking against Gauss–Newton methods based on generalized Krylov subspaces ($GNKS$). The results show that restarted polynomial extrapolation on PGD/SGD often yields faster convergence and lower reconstruction error than GNKS, especially in well-conditioned Bratu settings and very large sparse problems, while memory-conscious extrapolation remains practical. This work highlights the potential of extrapolation-accelerated preconditioned gradient methods as a robust alternative for large-scale nonlinear least-squares problems.

Abstract

Vector extrapolation methods are widely used in large-scale simulation studies, and numerous extrapolation-based acceleration techniques have been developed to enhance the convergence of linear and nonlinear fixed-point iterative methods. While classical extrapolation strategies often reduce the number of iterations or the computational cost, they do not necessarily lead to a significant improvement in the accuracy of the computed approximations. In this paper, we study the combination of preconditioned gradient-based methods with extrapolation strategies and propose an extrapolation-accelerated framework that simultaneously improves convergence and approximation accuracy. The focus is on the solution of nonlinear least-squares problems through the integration of vector extrapolation techniques with preconditioned gradient descent methods. A comprehensive set of numerical experiments is carried out to study the behavior of polynomial-type extrapolation methods and the vector $\varepsilon$-algorithm when coupled with gradient descent schemes, with and without preconditioning. The results demonstrate the impact of extrapolation techniques on both convergence rate and solution accuracy, and report iteration counts, computational times, and relative reconstruction errors. The performance of the proposed hybrid approaches is further assessed through a benchmarking study against Gauss--Newton methods based on generalized Krylov subspaces.

A New Combination of Preconditioned Gradient Descent Methods and Vector Extrapolation Techniques for Nonlinear Least-Squares Problems

TL;DR

The paper addresses the slow convergence of nonlinear least-squares solvers by coupling vector extrapolation techniques (RRE, MPE, VEA) with preconditioned gradient descent (PGD, SGD). It introduces restarted extrapolation frameworks and applies them to solve large-scale problems, notably the Bratu equation and an extremely sparse nonlinear model, benchmarking against Gauss–Newton methods based on generalized Krylov subspaces (). The results show that restarted polynomial extrapolation on PGD/SGD often yields faster convergence and lower reconstruction error than GNKS, especially in well-conditioned Bratu settings and very large sparse problems, while memory-conscious extrapolation remains practical. This work highlights the potential of extrapolation-accelerated preconditioned gradient methods as a robust alternative for large-scale nonlinear least-squares problems.

Abstract

Vector extrapolation methods are widely used in large-scale simulation studies, and numerous extrapolation-based acceleration techniques have been developed to enhance the convergence of linear and nonlinear fixed-point iterative methods. While classical extrapolation strategies often reduce the number of iterations or the computational cost, they do not necessarily lead to a significant improvement in the accuracy of the computed approximations. In this paper, we study the combination of preconditioned gradient-based methods with extrapolation strategies and propose an extrapolation-accelerated framework that simultaneously improves convergence and approximation accuracy. The focus is on the solution of nonlinear least-squares problems through the integration of vector extrapolation techniques with preconditioned gradient descent methods. A comprehensive set of numerical experiments is carried out to study the behavior of polynomial-type extrapolation methods and the vector -algorithm when coupled with gradient descent schemes, with and without preconditioning. The results demonstrate the impact of extrapolation techniques on both convergence rate and solution accuracy, and report iteration counts, computational times, and relative reconstruction errors. The performance of the proposed hybrid approaches is further assessed through a benchmarking study against Gauss--Newton methods based on generalized Krylov subspaces.
Paper Structure (14 sections, 2 theorems, 44 equations, 7 figures, 5 tables, 6 algorithms)

This paper contains 14 sections, 2 theorems, 44 equations, 7 figures, 5 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Vector extrapolation methods
  3. Polynomial extrapolation methods
  4. The vector $\varepsilon$-algorithm
  5. Solution of nonlinear least-squares problems
  6. Iterative methods
  7. An extrapolation-based method for nonlinear least-squares problems
  8. Numerical experiments
  9. Numerical study of the nonlinear Bratu problem
  10. Model problem and numerical setting
  11. Numerical study of the extended Bratu problem
  12. The standard Bratu problem
  13. An extremely sparse nonlinear problem
  14. Conclusion

Key Result

Proposition 1

Let $t_{k,q}$ be the transformation given by formula (eq:extrapolapprox). Then, the transformation exists and is unique if and only if $\mathrm{det}\left({ Y_{q}}^{T} \Delta^{2} S_{k,q} \right) \not = 0$.

Figures (7)

  • Figure 1: The vector $\varepsilon$-table.
  • Figure 2: Performance of gradient descent and its preconditioned versions with and without polynomial extrapolation.
  • Figure 3: Convergence of gradient descent and its preconditioned versions with and without polynomial extrapolation in comparison with Gauss--Newton using generalized Krylov subspaces.
  • Figure 4: Efficiency comparison between MPE, RRE, and VEA for the acceleration of the preconditioned gradient descent.
  • Figure 5: Performance comparison of the different algorithms for increasing dimensions of the 2D Bratu problem with various values of $\alpha$ and $\lambda$.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Proposition 1
  • Proposition 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4