Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On finite termination of quasi-Newton methods on quadratic problems

Aban Ansari-Önnestam, Anders Forsgren

TL;DR

The paper investigates finite termination of quasi-Newton methods for unconstrained quadratic problems with $H \succ 0$ by relaxing the need for exact line search and full conjugacy. It develops a subspace Newton framework within Krylov subspaces $\mathcal{K}_k(g_0,H)$ and shows that a memoryless quasi-Newton matrix acting on at most two vectors suffices to compute a Newton direction in a finite number of iterations, independent of step lengths. By constructing $B_k$ to mimic $H$ on a two-dimensional subspace, the approach yields a quasi-Newton step of the form $p_k(x)=q_k+p^N_{k-1}(x)$ that preserves finite termination, with a special case reducing to memoryless BFGS under exact line search. The authors also provide a first-order variant that uses gradient differences to infer the needed subspace action, and present numerical support demonstrating the predicted termination behavior. Overall, the work deepens the theoretical understanding of quasi-Newton methods on quadratics and suggests a pathway to finite-term, step-size-agnostic algorithms that could inform nonlinear extensions.

Abstract

Quasi-Newton methods form an important class of methods for solving nonlinear optimization problems. In such methods, first order information is used to approximate the second derivative. The aim is to mimic the fast convergence that can be guaranteed by Newton-based methods. In the best case, quasi-Newton methods will far outperform steepest descent and other first order methods, without the computational cost of calculating the exact second derivative. These convergence guarantees hold locally, which follows closely from the fact that, if the objective function is strongly convex, it can be approximated well by a quadratic function close to the solution. Understanding the performance of quasi-Newton methods on quadratic problems with a symmetric positive definite Hessian is therefore of vital importance. In the classic case, an approximation of the Hessian is updated at every iteration and exact line search is used. It is well known that the algorithm terminates finitely, even when the Hessian approximation is memoryless, i.e. requires only the most recent information. In this paper, we explore the possibilities in which reliance on exact line search and dependence on conjugate search directions can be relaxed, while preserving finite termination properties of quasi-Newton methods on quadratic problems. We show that it suffices to create a memoryless quasi-Newton matrix based on two vectors to give ability to compute a Newton direction within a finite number of iterations, independent of step lengths. It is unnecessary for the quasi-Newton approximation to act as the Hessian on the full space.

On finite termination of quasi-Newton methods on quadratic problems

TL;DR

The paper investigates finite termination of quasi-Newton methods for unconstrained quadratic problems with by relaxing the need for exact line search and full conjugacy. It develops a subspace Newton framework within Krylov subspaces and shows that a memoryless quasi-Newton matrix acting on at most two vectors suffices to compute a Newton direction in a finite number of iterations, independent of step lengths. By constructing to mimic on a two-dimensional subspace, the approach yields a quasi-Newton step of the form that preserves finite termination, with a special case reducing to memoryless BFGS under exact line search. The authors also provide a first-order variant that uses gradient differences to infer the needed subspace action, and present numerical support demonstrating the predicted termination behavior. Overall, the work deepens the theoretical understanding of quasi-Newton methods on quadratics and suggests a pathway to finite-term, step-size-agnostic algorithms that could inform nonlinear extensions.

Abstract

Quasi-Newton methods form an important class of methods for solving nonlinear optimization problems. In such methods, first order information is used to approximate the second derivative. The aim is to mimic the fast convergence that can be guaranteed by Newton-based methods. In the best case, quasi-Newton methods will far outperform steepest descent and other first order methods, without the computational cost of calculating the exact second derivative. These convergence guarantees hold locally, which follows closely from the fact that, if the objective function is strongly convex, it can be approximated well by a quadratic function close to the solution. Understanding the performance of quasi-Newton methods on quadratic problems with a symmetric positive definite Hessian is therefore of vital importance. In the classic case, an approximation of the Hessian is updated at every iteration and exact line search is used. It is well known that the algorithm terminates finitely, even when the Hessian approximation is memoryless, i.e. requires only the most recent information. In this paper, we explore the possibilities in which reliance on exact line search and dependence on conjugate search directions can be relaxed, while preserving finite termination properties of quasi-Newton methods on quadratic problems. We show that it suffices to create a memoryless quasi-Newton matrix based on two vectors to give ability to compute a Newton direction within a finite number of iterations, independent of step lengths. It is unnecessary for the quasi-Newton approximation to act as the Hessian on the full space.
Paper Structure (11 sections, 9 theorems, 57 equations, 1 table, 1 algorithm)

This paper contains 11 sections, 9 theorems, 57 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Generation of conjugate search directions using exact line search
  4. Quasi-Newton methods
  5. Subspace Newton method
  6. Subspace quasi-Newton methods
  7. An exact quasi-Newton method without exact line search
  8. Numerical support
  9. A first order formulation of Algorithm \ref{['alg:main']}
  10. Conclusion
  11. Appendix: Some linear algebra results

Key Result

Lemma 3.1

For $k \leq r$, let $q_0, \dots, q_{k-1}$ be a basis for $\mathcal{K}_k(g_0,H)$ where $q_i // \hat{q}_i$. For any $x \in x_0 + \mathcal{K}_k(g_0,H)$, the subspace Newton step $p^N_{k-1}(x)$ to $\hat{x}_k$ is Additionally, for any $\alpha \in \mathbb{R}$

Theorems & Definitions (11)

  • Definition 2.1
  • Definition 2.2
  • Lemma 3.1
  • Proposition 3.1
  • Lemma 4.1
  • Proposition 4.1
  • Theorem 5.1
  • Corollary 5.1
  • Lemma 5.1
  • Lemma A.1
  • ...and 1 more