Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extended-Krylov-subspace methods for trust-region and norm-regularization subproblems

Hussam Al Daas, Nicholas I. M. Gould

TL;DR

This work addresses efficient resolution of trust-region and norm-regularization subproblems by revealing that the solution manifold $\{x(\sigma)\}$ across parameter variations lies on an approximately low-rank structure. It introduces an extended-Krylov-subspace framework (TREK for trust-region, NREK for regularization) that builds a compact basis via a single matrix factorization and reduces subproblems to small, diagonalizable systems solvable with high-order root-finding. A convergence analysis shows linear-at-worst convergence of the extended Krylov basis to the solution manifold, underpinning the practical efficiency observed in large-scale experiments on CUTEst problems. The approach offers a compelling alternative to multi-factorization and purely factorization-free methods, with practical GALAHAD implementations supporting sequences of subproblems and elliptical-norm generalizations.

Abstract

We consider an effective new method for solving trust-region and norm-regularization problems that arise as subproblems in many optimization applications. We show that the solutions to such subproblems lie on a manifold of approximately very low rank as a function of their controlling parameters (trust-region radius or regularization weight). Based on this, we build a basis for this manifold using an efficient extended-Krylov-subspace iteration that involves a single matrix factorization. The problems within the subspace using such a basis may be solved at very low cost using effective high-order root-finding methods. This then provides an alternative to common methods using multiple factorizations or standard Krylov subspaces. We provide numerical results to illustrate the effectiveness of our {\tt TREK}/{\tt NREK} approach.

Extended-Krylov-subspace methods for trust-region and norm-regularization subproblems

TL;DR

This work addresses efficient resolution of trust-region and norm-regularization subproblems by revealing that the solution manifold across parameter variations lies on an approximately low-rank structure. It introduces an extended-Krylov-subspace framework (TREK for trust-region, NREK for regularization) that builds a compact basis via a single matrix factorization and reduces subproblems to small, diagonalizable systems solvable with high-order root-finding. A convergence analysis shows linear-at-worst convergence of the extended Krylov basis to the solution manifold, underpinning the practical efficiency observed in large-scale experiments on CUTEst problems. The approach offers a compelling alternative to multi-factorization and purely factorization-free methods, with practical GALAHAD implementations supporting sequences of subproblems and elliptical-norm generalizations.

Abstract

We consider an effective new method for solving trust-region and norm-regularization problems that arise as subproblems in many optimization applications. We show that the solutions to such subproblems lie on a manifold of approximately very low rank as a function of their controlling parameters (trust-region radius or regularization weight). Based on this, we build a basis for this manifold using an efficient extended-Krylov-subspace iteration that involves a single matrix factorization. The problems within the subspace using such a basis may be solved at very low cost using effective high-order root-finding methods. This then provides an alternative to common methods using multiple factorizations or standard Krylov subspaces. We provide numerical results to illustrate the effectiveness of our {\tt TREK}/{\tt NREK} approach.

Paper Structure

This paper contains 15 sections, 3 theorems, 104 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Cauchy matrices and accurate low-rank approximation
  3. Manifold approximation from Krylov subspaces
  4. The EKS algorithm
  5. Solving the convex trust-region subproblem
  6. Resolves
  7. Other elliptical norms
  8. Solving the general trust-region subproblem
  9. Other elliptical norms
  10. Numerical experiments
  11. Regularization subproblems
  12. Conclusions
  13. Convergence analysis
  14. Modified norm trust region subproblem solver
  15. Detailed numerical experiments

Key Result

Proposition 1

Given $A$, $\Phi$, $\Psi$, $\phi_k$, $M_k$ and ${\@fontswitch{}{\mathcal{}} W}$ as above, we have that

Figures (2)

  • Figure 7.1: Performance profile comparing three trust-region subproblem solvers on larger CUTEst examples.
  • Figure A.1: Upper bound on the convergence rate of extended Krylov method to solve the shifted linear system with a shift $\sigma$. The spectrum of the matrix is in $[5,100]$.

Theorems & Definitions (6)

  • Proposition 1
  • Proof 1
  • Proposition 2
  • Proof 2
  • Theorem 3
  • Proof 3