Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient gradient-based methods for bilevel learning via recycling Krylov subspaces

Matthias J. Ehrhardt, Silvia Gazzola, Sebastian J. Scott

TL;DR

This work tackles the high cost of computing hypergradients in bilevel learning by introducing recycling Krylov subspaces that reuse information across successive Hessian solves. It advances a bilevel-aware recycling strategy based on the generalized singular value decomposition (GSVD), using Ritz generalized singular vectors to build recycle spaces that account for the premultiplication by the Jacobian in hypergradients. A new stopping criterion that directly approximates hypergradient error is proposed, enabling early termination without sacrificing gradient quality. Numerical experiments on imaging inverse problems (e.g., MNIST inpainting and BSDS300 deconvolution) demonstrate substantial reductions in Hessian-solve iterations and overall computation while preserving hypergradient accuracy, highlighting the method’s practical impact for scalable bilevel learning in large-scale inverse problems.

Abstract

Many optimization problems require hyperparameters, i.e., parameters that must be pre-specified in advance, such as regularization parameters and parametric regularizers in variational regularization methods for inverse problems, and dictionaries in compressed sensing. A data-driven approach to determine appropriate hyperparameter values is via a nested optimization framework known as bilevel learning. Even when it is possible to employ a gradient-based solver to the bilevel optimization problem, construction of the gradients, known as hypergradients, is computationally challenging, each one requiring both a solution of a minimization problem and a linear system solve. These systems do not change much during the iterations, which motivates us to apply recycling Krylov subspace methods, wherein information from one linear system solve is re-used to solve the next linear system. Existing recycling strategies often employ eigenvector approximations called Ritz vectors. In this work we propose a novel recycling strategy based on a new concept, Ritz generalized singular vectors, which acknowledge the bilevel setting. Additionally, while existing iterative methods primarily terminate according to the residual norm, this new concept allows us to define a new stopping criterion that directly approximates the error of the associated hypergradient. The proposed approach is validated through extensive numerical testing in the context of inverse problems in imaging.

Efficient gradient-based methods for bilevel learning via recycling Krylov subspaces

TL;DR

This work tackles the high cost of computing hypergradients in bilevel learning by introducing recycling Krylov subspaces that reuse information across successive Hessian solves. It advances a bilevel-aware recycling strategy based on the generalized singular value decomposition (GSVD), using Ritz generalized singular vectors to build recycle spaces that account for the premultiplication by the Jacobian in hypergradients. A new stopping criterion that directly approximates hypergradient error is proposed, enabling early termination without sacrificing gradient quality. Numerical experiments on imaging inverse problems (e.g., MNIST inpainting and BSDS300 deconvolution) demonstrate substantial reductions in Hessian-solve iterations and overall computation while preserving hypergradient accuracy, highlighting the method’s practical impact for scalable bilevel learning in large-scale inverse problems.

Abstract

Many optimization problems require hyperparameters, i.e., parameters that must be pre-specified in advance, such as regularization parameters and parametric regularizers in variational regularization methods for inverse problems, and dictionaries in compressed sensing. A data-driven approach to determine appropriate hyperparameter values is via a nested optimization framework known as bilevel learning. Even when it is possible to employ a gradient-based solver to the bilevel optimization problem, construction of the gradients, known as hypergradients, is computationally challenging, each one requiring both a solution of a minimization problem and a linear system solve. These systems do not change much during the iterations, which motivates us to apply recycling Krylov subspace methods, wherein information from one linear system solve is re-used to solve the next linear system. Existing recycling strategies often employ eigenvector approximations called Ritz vectors. In this work we propose a novel recycling strategy based on a new concept, Ritz generalized singular vectors, which acknowledge the bilevel setting. Additionally, while existing iterative methods primarily terminate according to the residual norm, this new concept allows us to define a new stopping criterion that directly approximates the error of the associated hypergradient. The proposed approach is validated through extensive numerical testing in the context of inverse problems in imaging.

Paper Structure

This paper contains 16 sections, 1 theorem, 42 equations, 14 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Gradient-based solvers for bilevel optimization
  4. (Recycling) Krylov methods
  5. MINRES
  6. RMINRES
  7. Defining and computing the recycle space for bilevel optimization
  8. Existing selection criteria
  9. Acknowledging the bilevel setting
  10. Alternative stopping criterion for \ref{['eq:hess_seq']}
  11. Numerical experiments
  12. MNIST Inpainting
  13. BSDS300 Deconvolution
  14. Conclusions and outlook
  15. Calculating the residual vector for RMINRES
  16. ...and 1 more sections

Key Result

Theorem 3.1

\newlabelthm:gsvd0 Let $\tilde{H}\in\mathbb{R}^{t\times t}$ be invertible and $\tilde{ J}\in\mathbb{R}^{p\times t}$ with $p\geq t.$ There exist orthogonal $V_{\tilde{ J}} \in\mathbb{R}^{p\times p}$ and $V_{\tilde{H}}\in\mathbb{R}^{t\times t}$ and invertible $X\in\mathbb{R}^{t\times t}$ such that where $0\leq \alpha_{1} \leq \ldots \leq \alpha_t< 1$ and $1\geq \beta_{1} \geq \ldots \geq \beta_t>

Figures (14)

  • Figure 1: Data and reconstructions for the inpainting problem. We see that the Fields of Experts regularizer with optimized parameters can provide a much better reconstruction than TV.
  • Figure 2: Breakdown of timings for solving the inpainting bilevel problem. The titles of the subfigures in the top row indicate the total time (seconds) and percentage contribution towards the total computation time. Each entry in the plot of timings for lower level solves is the sum of both the time for the initial solve and timings of all solves performed within backtracking linesearch for that iteration of gradient descent for the upper level problem. Over 26% of computation time is spent solving the sequence of Hessian systems.
  • Figure 3: Relative differences in the Frobenius norm for the sequence of, from left to right: the Hessians, right hand sides, and determined solutions of the linear systems. Since relative differences of order $10^{-2}$ are observed, recycling may provide a viable approach to speed up computations.
  • Figure 4: Comparison between solving the sequence of Hessians without recycling using conjugate gradient (CG) and MINRES. Both methods have a similar performance, with MINRES terminating earlier and the hypergradients associated to CG being slightly higher quality.
  • Figure 5: Comparison between the performance of Ritz vectors to the eigenvectors they approximate, and Ritz generalized singular vectors to generalized singular vectors. A summary of the acronyms appearing in the legends is given in Table \ref{['tab:acro']}.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Theorem 3.1: GSVD