Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residual

Guangyong Chen, Peng Xue, Min Gan, Jing Chen, Wenzhong Guo, C. L. Philip. Chen

TL;DR

This work analyzes variable projection ($SNLLS$) for separable nonlinear models, establishing a theoretical framework that links Jacobian-approximation choices to local convergence. It proves that Kaufman’s simplified Jacobian can achieve a local convergence rate comparable to Golub & Pereyra’s form under standard regularity conditions, and introduces VPLR, a Hessian-corrected VP variant equipped with a quasi-Newton–like update to handle large residuals. Numerical experiments on synthetic exponential fitting, RBF-AR time-series forecasting, and RBF-network fitting demonstrate faster convergence and improved accuracy for VP and especially VPLR compared with Joint, AM, and BCD approaches. Overall, the paper strengthens theoretical understanding of VP and provides a practically effective method for challenging large-residual separable nonlinear optimization problems.

Abstract

This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification, image processing, and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman's VP algorithm can achieve a similar convergence rate as the Golub \& Pereyra's form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, drawing inspiration from these theoretical revelations, we design a refined VP algorithm for handling separable nonlinear optimization problems characterized by large residual, called VPLR, which boosts the convergence performance by addressing the interdependence of parameters within the separable model and by continually correcting the approximated Hessian matrix to counteract the influence of large residual during the iterative process. The effectiveness of this refined algorithm is corroborated through numerical experimentation.

Variable Projection Algorithms: Theoretical Insights and A Novel Approach for Problems with Large Residual

TL;DR

This work analyzes variable projection () for separable nonlinear models, establishing a theoretical framework that links Jacobian-approximation choices to local convergence. It proves that Kaufman’s simplified Jacobian can achieve a local convergence rate comparable to Golub & Pereyra’s form under standard regularity conditions, and introduces VPLR, a Hessian-corrected VP variant equipped with a quasi-Newton–like update to handle large residuals. Numerical experiments on synthetic exponential fitting, RBF-AR time-series forecasting, and RBF-network fitting demonstrate faster convergence and improved accuracy for VP and especially VPLR compared with Joint, AM, and BCD approaches. Overall, the paper strengthens theoretical understanding of VP and provides a practically effective method for challenging large-residual separable nonlinear optimization problems.

Abstract

This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification, image processing, and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman's VP algorithm can achieve a similar convergence rate as the Golub \& Pereyra's form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, drawing inspiration from these theoretical revelations, we design a refined VP algorithm for handling separable nonlinear optimization problems characterized by large residual, called VPLR, which boosts the convergence performance by addressing the interdependence of parameters within the separable model and by continually correcting the approximated Hessian matrix to counteract the influence of large residual during the iterative process. The effectiveness of this refined algorithm is corroborated through numerical experimentation.
Paper Structure (10 sections, 4 theorems, 65 equations, 8 figures, 2 tables, 1 algorithm)

This paper contains 10 sections, 4 theorems, 65 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Convergence Analysis of Variable Projection Algorithm
  3. Different forms of VP algorithms
  4. Comparison of VP Algorithm and Joint Algorithm
  5. Variable Projection for Separable Nonlinear Problem with Large Residual
  6. NUMERICAL EXAMPLES
  7. Parameter Estimation of Exponential Model
  8. Forecasting of Nonlinear Time Series Using the RBF-AR Model
  9. Fitting Concrete Data Using the RBF Network
  10. CONCLUSIONS

Key Result

Lemma 1

Suppose that the Jacobian matrix $\mathbf{J}(\boldsymbol{a})$ is Lipschitz continuous on the bounded set $\mathcal{B}({\color{black} \boldsymbol{a}^*},{\color{black} \rho})$, i.e., there exists a constant $L>0$ such that $\left\|\mathbf{J}(\boldsymbol{a}_1) - \mathbf{J}(\boldsymbol{a}_2)\right\| \le

Figures (8)

  • Figure 1: Observational data generated from the exponential model
  • Figure 2: Comparative convergence of different algorithms during iterative process.
  • Figure 3: Comparison of the results of fitting observed data to models estimated by different algorithms.
  • Figure 4: Performance comparison of different algorithms with 100 sets of randomized tests.
  • Figure 5: Comparison of execution times across different algorithms using 100 sets of randomized tests.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof