Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grassmannian Geometry and Global Convergence of Variable Projection for Neural Networks

Mathias Dus

TL;DR

This work develops a geometric VarPro framework for neural networks by embedding variable projection in the Grassmannian. By eliminating linear output weights, it reduces training to optimization over subspaces on Gr(n,p) and provides explicit expressions for the Riemannian gradient and Hessian, establishing a benign landscape with no spurious local minima under overparameterization. It further introduces a regularized projector manifold to handle rank-deficient feature maps, enabling continuous optimization across strata and preserving convergence guarantees. The approach yields practical benefits for regression and PINN problems and includes an efficient heat equation solver, highlighting the method's stability and scalability. Overall, the paper bridges classical SNLS theory with modern feature learning, offering a principled, geometry-driven route to robust convergence in physics-informed and deep learning contexts.

Abstract

Training deep neural networks and Physics-Informed Neural Networks (PINNs) often leads to ill-conditioned and stiff optimization problems. A key structural feature of these models is that they are linear in the output-layer parameters and nonlinear in the hiddenlayer parameters, yielding a separable nonlinear least-squares formulation. In this work, we study the classical variable projection (VarPro) method for such problems in the context of deep neural networks. We provide a geometric formulation on the Grassmannian and analyze the structure of critical points and convergence properties of the reduced problem. When the feature map is parametrized by a neural network, we show that these properties persist except in rank-deficient regimes, which we address via a regularized Grassmannian framework. Numerical experiments for regression and PINNs, including an efficient solver for the heat equation, illustrate the practical effectiveness of the approach.

Grassmannian Geometry and Global Convergence of Variable Projection for Neural Networks

TL;DR

This work develops a geometric VarPro framework for neural networks by embedding variable projection in the Grassmannian. By eliminating linear output weights, it reduces training to optimization over subspaces on Gr(n,p) and provides explicit expressions for the Riemannian gradient and Hessian, establishing a benign landscape with no spurious local minima under overparameterization. It further introduces a regularized projector manifold to handle rank-deficient feature maps, enabling continuous optimization across strata and preserving convergence guarantees. The approach yields practical benefits for regression and PINN problems and includes an efficient heat equation solver, highlighting the method's stability and scalability. Overall, the paper bridges classical SNLS theory with modern feature learning, offering a principled, geometry-driven route to robust convergence in physics-informed and deep learning contexts.

Abstract

Training deep neural networks and Physics-Informed Neural Networks (PINNs) often leads to ill-conditioned and stiff optimization problems. A key structural feature of these models is that they are linear in the output-layer parameters and nonlinear in the hiddenlayer parameters, yielding a separable nonlinear least-squares formulation. In this work, we study the classical variable projection (VarPro) method for such problems in the context of deep neural networks. We provide a geometric formulation on the Grassmannian and analyze the structure of critical points and convergence properties of the reduced problem. When the feature map is parametrized by a neural network, we show that these properties persist except in rank-deficient regimes, which we address via a regularized Grassmannian framework. Numerical experiments for regression and PINNs, including an efficient solver for the heat equation, illustrate the practical effectiveness of the approach.
Paper Structure (52 sections, 27 theorems, 148 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 52 sections, 27 theorems, 148 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Litterature review
  3. The Stiefel and the Grassmannian manifolds
  4. The Tangent Space
  5. Inclusion ($\supseteq$)
  6. Surjectivity via Dimensional Analysis
  7. The Grassmannian as a Quotient
  8. Vertical and Horizontal Decomposition
  9. The Vertical Space
  10. Inclusion ($\supseteq$)
  11. Inclusion ($\subseteq$)
  12. The Horizontal Space
  13. Characterization of the Grassmannian Tangent Space
  14. Geodesics and Second-Order Approximation
  15. Geodesics via Horizontal Lifts
  16. ...and 37 more sections

Key Result

Lemma 2.1

The tangent space to $St(n,p)$ at a point $X$ is given by: Equivalently, $Z \in T_X St(n,p)$ if and only if $X^\top Z$ is a skew-symmetric matrix.

Figures (7)

  • Figure 1: The fiber bundle $St(n,p)$
  • Figure 2: Results of VarPro on some regression problems
  • Figure 3: Case $d=10$, $f(x) = \sum_{i=1}^d \sin(2 \pi x_i)$
  • Figure 4: The case $d=1$ with $\varepsilon= 10^{-16}, f(x) = \sin(8 \pi x)$
  • Figure 5: Results for some Poisson's problems
  • ...and 2 more figures

Theorems & Definitions (66)

  • Definition 2.1
  • Lemma 2.1: Characterization of the Stiefel Tangent Space
  • proof
  • Proposition 2.2
  • proof
  • Lemma 2.3
  • proof
  • Theorem 2.4: Vertical Space Characterization
  • proof
  • Theorem 2.5: Horizontal Space Characterization
  • ...and 56 more