Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Analysis of a Variable Projection Method for Regularized Separable Nonlinear Inverse Problems

Malena I. Español, Gabriela Jeronimo

TL;DR

This work addresses regularized separable nonlinear inverse problems where the forward operator $A({\bf y})$ depends on a small set of nonlinear parameters ${\bf y}$ and the solution ${\bf x}$ is high-dimensional. The authors leverage the Variable Projection framework to reduce the problem to a nonlinear least-squares optimization in ${\bf y}$ and apply Gauss-Newton, while recognizing the high cost of exact Jacobians; they introduce GenVarPro and its inexact variant, Inexact-GenVarPro, which uses LSQR to approximate inner solves and Jacobians with a principled stopping criterion. A rigorous convergence analysis is developed, providing inner-solver bounds and a main result that ensures convergence to the optimal ${\bf y}^*$ under Lipschitz and conditioning assumptions when the tolerance sequence $\{\varepsilon^{(k)}\}$ decays appropriately. Numerical experiments on a blind deconvolution problem corroborate the theory, showing that the exponentially decaying LSQR tolerance yields fast, robust convergence in practice. The proposed approach enables scalable, precise solutions for large-scale regularized inverse problems by balancing inner-iteration accuracy with outer Gauss-Newton progress, with potential extensions to alternative inner solvers and adaptive regularization parameter strategies.

Abstract

Variable projection methods prove highly efficient in solving separable nonlinear least squares problems by transforming them into a reduced nonlinear least squares problem, typically solvable via the Gauss-Newton method. When solving large-scale separable nonlinear inverse problems with general-form Tikhonov regularization, the computational demand for computing Jacobians in the Gauss-Newton method becomes very challenging. To mitigate this, iterative methods, specifically LSQR, can be used as inner solvers to compute approximate Jacobians. This article analyzes the impact of these approximate Jacobians within the variable projection method and introduces stopping criteria to ensure convergence. We also present numerical experiments where we apply the proposed method to solve a blind deconvolution problem to illustrate and confirm our theoretical results.

Convergence Analysis of a Variable Projection Method for Regularized Separable Nonlinear Inverse Problems

TL;DR

This work addresses regularized separable nonlinear inverse problems where the forward operator depends on a small set of nonlinear parameters and the solution is high-dimensional. The authors leverage the Variable Projection framework to reduce the problem to a nonlinear least-squares optimization in and apply Gauss-Newton, while recognizing the high cost of exact Jacobians; they introduce GenVarPro and its inexact variant, Inexact-GenVarPro, which uses LSQR to approximate inner solves and Jacobians with a principled stopping criterion. A rigorous convergence analysis is developed, providing inner-solver bounds and a main result that ensures convergence to the optimal under Lipschitz and conditioning assumptions when the tolerance sequence decays appropriately. Numerical experiments on a blind deconvolution problem corroborate the theory, showing that the exponentially decaying LSQR tolerance yields fast, robust convergence in practice. The proposed approach enables scalable, precise solutions for large-scale regularized inverse problems by balancing inner-iteration accuracy with outer Gauss-Newton progress, with potential extensions to alternative inner solvers and adaptive regularization parameter strategies.

Abstract

Variable projection methods prove highly efficient in solving separable nonlinear least squares problems by transforming them into a reduced nonlinear least squares problem, typically solvable via the Gauss-Newton method. When solving large-scale separable nonlinear inverse problems with general-form Tikhonov regularization, the computational demand for computing Jacobians in the Gauss-Newton method becomes very challenging. To mitigate this, iterative methods, specifically LSQR, can be used as inner solvers to compute approximate Jacobians. This article analyzes the impact of these approximate Jacobians within the variable projection method and introduces stopping criteria to ensure convergence. We also present numerical experiments where we apply the proposed method to solve a blind deconvolution problem to illustrate and confirm our theoretical results.
Paper Structure (9 sections, 3 theorems, 76 equations, 5 figures, 3 tables, 2 algorithms)

This paper contains 9 sections, 3 theorems, 76 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Variable Projection Methods for Regularized Problems
  3. GenVarPro
  4. Inexact-GenVarPro
  5. Convergence Analysis
  6. Inner solver bounds
  7. Main result
  8. Numerical Example
  9. Conclusions

Key Result

Lemma 3.1

\newlabellem:approx_solution_M0 Let ${\bf M} \in \mathbb{R}^{l\times n}$, with $l\ge n$, be a matrix of full rank, ${\bf d} \in \mathbb{R}^{l\times 1}$, and ${\bf x}\in \mathbb{R}^{n\times 1}$ the solution of the problem For $\varepsilon >0$ such that $\kappa_2({\bf M}) \varepsilon<1$, let $\bar{{\bf x}}$ be an approximation of ${\bf x}$ computed using an iterative method with stopping criterion

Figures (5)

  • Figure 1: Left: the true signal ${\bf x}_{{\rm true}}$ and the blurred and noisy signal ${\bf b}$. Right: a reconstructed Tikhonov solution using ${\bf y}_{{\rm true}}$.
  • Figure 2: Convergence curves of the GenVarPro (GP) and Inexact-GenVarPro method with different tolerances: 1) $\varepsilon^{(k)}=\varepsilon^{(0)}$(LSQR-b), 2) $\varepsilon^{(k)}=\varepsilon^{(0)}/k$(LSQR-lb), 3) $\varepsilon^{(k)}=\varepsilon^{(k-1)}/2$(LSQR-ab), and 4) $\varepsilon^{(k)} = 10^{-11}$(LSQR-s). The left column contains the values of ${\bf y}^{(k)}$ for each iteration obtained using ${\bf y}^{(0)}=2$ (top) and ${\bf y}^{(0)}=4$ (bottom). The right column depicts the values of the function $\mathcal{F}(\bar{{\bf x}}^{(k)}, {\bf y}^{(k)})$ for each iteration using ${\bf y}^{(0)}=2$ (top) and ${\bf y}^{(0)}=4$ (bottom).
  • Figure 3: Distances between the solutions ${\bf y}$ given at each iteration by GenVarPro (${\bf y}^{(k)}_{\rm{GP}}$) and Inexact-GenVarPro (${\bf y}^{(k)}_{\rm{LSQR}}$) with different tolerances (LSQR-b, LSQR-lb, LSQR-ab, and LSQR-s) using ${\bf y}^{(0)}=2$ (top) and ${\bf y}^{(0)}=4$ (bottom). The right column contains the same data using a logarithmic scale in the vertical axis.
  • Figure 4: Left column: $\|{\bf x}^{(k)}-\bar{{\bf x}}^{(k)}\|_2$, where ${\bf x}^{(k)}$ is the exact solution of the linear subproblem and $\bar{{\bf x}}^{(k)}$ its LSQR approximation at each iteration, using LSQR-b, LSQR-lb, LSQR-ab, and LSQR-s, respectively, with ${\bf y}^{(0)}=2$ (top) and ${\bf y}^{(0)}=4$ (bottom). Right column: error bounds computed according to \ref{['eq:approx_solution']} for each case.
  • Figure 5: Reconstructions of the solution ${\bf x}$ computed by GenVarPro and Inexact-GenVarPor using LSQR with an exponential decreasing tolerance (LSQR-ab) for ${\bf y}^{(0)}=2$ (left) and ${\bf y}^{(0)}=4$ (right) after seven iterations.

Theorems & Definitions (7)

  • Lemma 3.1
  • Proof 1
  • Lemma 3.2
  • Proof 2
  • Theorem 3.3
  • Proof 3
  • Remark 3.4