Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variable projection framework for the reduced-rank matrix approximation problem by weighted least-squares

Pascal Terray

TL;DR

This work analyzes the weighted low-rank approximation (WLRA) problem and develops a variable projection framework for efficient, robust, and scalable optimization. It establishes the mathematical foundation across linear algebra, multilinear algebra, topology, and differential geometry, and then connects WLRA to separable nonlinear least-squares formulations that leverage a bilinear factorization $\mathbf{Y}=\mathbf{A}\mathbf{B}$. The paper proves key equivalences and detailed optimality conditions between the original WLRA formulations and their factorized/VP forms, discusses regularized and approximate variants (e.g., nuclear-norm relaxations, MMMF, continuation methods), and presents practical algorithms based on block alternating least-squares (ALS/NIPALS) and variable-projection techniques. These results illuminate how WLRA can be solved with modern second-order methods and Riemannian optimization on Grassmann manifolds, offering insight into solvability, landscape, and non-smoothness, with implications for robust matrix completion, denoising, and low-rank recovery in diverse domains.

Abstract

In this monograph, we review and develop variable projection Gauss-Newton, Levenberg-Marquardt and Newton methods for the Weighted Low-Rank Approximation (WLRA) problem, which has now an increasing number of applications in many scientific fields. Particular attention is drawn at the robustness, efficiency and scalability of these variable projection second-order algorithms such that they can be used also on larger datasets now commonly found in many practical problems for which only first-order algorithms based on sequential repetitions of local optimization (e.g., majorization, Expectation-Maximization or alternating least-squares methods) or variations of gradient descent (e.g., conjugate, proximal or stochastic gradient descent methods), or hybrid algorithms from these two classes of methods, were only feasible due to their lower cost and memory requirement per iteration. In parallel with this review of variable projection algorithms, we develop new formulae for the Jacobian and Hessian matrices involved in these variable projection methods and demonstrate their very specific properties such as the uniform rank deficiency of the Jacobian matrix or the rank deficiency of the Hessian matrix at the (local) minimizers of the cost function associated with the WLRA problem. These systematic deficiencies must be taken into account in any practical implementations of the algorithms. These different properties and the very particular geometry of the WLRA problem have not been well appreciated in the past and have been the main obstacles in the development of robust variable projection second-order algorithms for solving the WLRA problem. In addition, we demonstrate that the variable projection framework gives original insights on the solvability, the landscape and the non-smoothness of the WLRA problem. It also helps to describe the tight links between previously unrelated methods, which have been proposed to solve it. Specifically, we illustrate the closed links between the variable projection framework and Riemannian optimization on the Grassmann manifold for the WLRA problem. We expect that software's developers and practitioners in different fields such as computer vision, signal processing, recommender systems, machine learning, multivariate statistics and geophysical sciences will benefit from the results in this monograph in order to devise more robust and accurate algorithms to solve the WLRA problem.

Variable projection framework for the reduced-rank matrix approximation problem by weighted least-squares

TL;DR

This work analyzes the weighted low-rank approximation (WLRA) problem and develops a variable projection framework for efficient, robust, and scalable optimization. It establishes the mathematical foundation across linear algebra, multilinear algebra, topology, and differential geometry, and then connects WLRA to separable nonlinear least-squares formulations that leverage a bilinear factorization . The paper proves key equivalences and detailed optimality conditions between the original WLRA formulations and their factorized/VP forms, discusses regularized and approximate variants (e.g., nuclear-norm relaxations, MMMF, continuation methods), and presents practical algorithms based on block alternating least-squares (ALS/NIPALS) and variable-projection techniques. These results illuminate how WLRA can be solved with modern second-order methods and Riemannian optimization on Grassmann manifolds, offering insight into solvability, landscape, and non-smoothness, with implications for robust matrix completion, denoising, and low-rank recovery in diverse domains.

Abstract

In this monograph, we review and develop variable projection Gauss-Newton, Levenberg-Marquardt and Newton methods for the Weighted Low-Rank Approximation (WLRA) problem, which has now an increasing number of applications in many scientific fields. Particular attention is drawn at the robustness, efficiency and scalability of these variable projection second-order algorithms such that they can be used also on larger datasets now commonly found in many practical problems for which only first-order algorithms based on sequential repetitions of local optimization (e.g., majorization, Expectation-Maximization or alternating least-squares methods) or variations of gradient descent (e.g., conjugate, proximal or stochastic gradient descent methods), or hybrid algorithms from these two classes of methods, were only feasible due to their lower cost and memory requirement per iteration. In parallel with this review of variable projection algorithms, we develop new formulae for the Jacobian and Hessian matrices involved in these variable projection methods and demonstrate their very specific properties such as the uniform rank deficiency of the Jacobian matrix or the rank deficiency of the Hessian matrix at the (local) minimizers of the cost function associated with the WLRA problem. These systematic deficiencies must be taken into account in any practical implementations of the algorithms. These different properties and the very particular geometry of the WLRA problem have not been well appreciated in the past and have been the main obstacles in the development of robust variable projection second-order algorithms for solving the WLRA problem. In addition, we demonstrate that the variable projection framework gives original insights on the solvability, the landscape and the non-smoothness of the WLRA problem. It also helps to describe the tight links between previously unrelated methods, which have been proposed to solve it. Specifically, we illustrate the closed links between the variable projection framework and Riemannian optimization on the Grassmann manifold for the WLRA problem. We expect that software's developers and practitioners in different fields such as computer vision, signal processing, recommender systems, machine learning, multivariate statistics and geophysical sciences will benefit from the results in this monograph in order to devise more robust and accurate algorithms to solve the WLRA problem.
Paper Structure (22 sections, 52 theorems, 1080 equations)

This paper contains 22 sections, 52 theorems, 1080 equations.

Table of Contents

  1. Introduction
  2. Definitions and preliminaries
  3. Linear algebra
  4. Multilinear algebra
  5. Topology of Euclidean vector or Frobenius matrix spaces
  6. Differential calculus, variational geometry and optimization
  7. Alternative and separable forms of the weighted low-rank approximation problem
  8. Nonconvex formulations of the WLRA problem
  9. Landscape connections of formulations \ref{['eq:P0']} and \ref{['eq:P1']} of the WLRA problem
  10. Approximate and regularized forms of the WLRA problem
  11. Variable projection formulation of the WLRA problem
  12. The block alternating least-squares method and its variants
  13. The variable projection framework
  14. Second-order NLLS optimization methods
  15. Computation and properties of the Jacobian matrix
  16. ...and 7 more sections

Key Result

Theorem 2.1

Let the SVD of $\mathbf{C} \in \mathbb{R}^{p \times n}$ be $\mathbf{C} = \mathbf{U}\Sigma\mathbf{V}^{T}$ with $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_{min(p,n)}$. In addition, for $k$ such that $1 \le k \le min(p,n)$, defined the truncated SVD of $\mathbf{C}$ by where $\mathbf{U}_{k}$ and $\mathbf{V}_{k}$ are the submatrices formed by the $k$ first columns of $\mathbf{U}$ and $\mathbf{V}$,

Theorems & Definitions (131)

  • Theorem 2.1
  • Lemma 2.2
  • proof
  • Definition 2.1
  • Definition 2.2
  • Theorem 2.3
  • proof
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • ...and 121 more