Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geodesics in the Deep Linear Network

Alan Chen

TL;DR

The paper studies geodesics in the DLN geometry by viewing the end-to-end map as a Riemannian submersion from a balanced manifold $\\mathcal{M}$ to the observable space $(\\mathfrak{M}_d, g^N)$. It derives general ODEs for geodesics on $(\\mathfrak{M}_d, g^N)$ and gives explicit closed-form DLN geodesics in the special case where the endpoints' singular vectors are related by a fixed orthogonal matrix $Q \\in O_d$, by lifting horizontal straight lines. The explicit geodesic formula is presented in terms of the SVD components and reduces to simpler diagonal forms in special limits, aligned with the Bures-Wasserstein case $N=2$ and its diagonal limit. The work lays a geometric foundation for understanding training dynamics in DLN via geodesic distances and suggests open problems for arbitrary endpoints and rank-deficient regimes.

Abstract

We derive a general system of ODEs and associated explicit solutions in a special case for geodesics between full rank matrices in the deep linear network geometry. In the process, we characterize all horizontal straight lines in the invariant balanced manifold that remain geodesics under Riemannian submersion.

Geodesics in the Deep Linear Network

TL;DR

The paper studies geodesics in the DLN geometry by viewing the end-to-end map as a Riemannian submersion from a balanced manifold to the observable space . It derives general ODEs for geodesics on and gives explicit closed-form DLN geodesics in the special case where the endpoints' singular vectors are related by a fixed orthogonal matrix , by lifting horizontal straight lines. The explicit geodesic formula is presented in terms of the SVD components and reduces to simpler diagonal forms in special limits, aligned with the Bures-Wasserstein case and its diagonal limit. The work lays a geometric foundation for understanding training dynamics in DLN via geodesic distances and suggests open problems for arbitrary endpoints and rank-deficient regimes.

Abstract

We derive a general system of ODEs and associated explicit solutions in a special case for geodesics between full rank matrices in the deep linear network geometry. In the process, we characterize all horizontal straight lines in the invariant balanced manifold that remain geodesics under Riemannian submersion.

Paper Structure

This paper contains 12 sections, 10 theorems, 74 equations.

Table of Contents

  1. Introduction
  2. The Deep Linear Network.
  3. Balancedness and Riemannian Geometry
  4. Main Results
  5. Proofs
  6. Geodesics Review
  7. The Bures-Wasserstein Geometry
  8. General Equations from Hamiltonian
  9. Explicit Solutions from Submersion
  10. Riemannian Geometry Background
  11. Open Problems
  12. Informal Note on Equation \ref{['eq:infinite_depth_geodesic']}

Key Result

Theorem 1.1

The end-to-end product map $\phi: \mathcal{M} \to \mathfrak{M}_d$ is a Riemannian submersion from $(\mathcal{M}, \iota)$ to $(\mathfrak{M}_d, g^N)$.

Theorems & Definitions (23)

  • Theorem 1.1: $\phi$ is a Riemannian Submersion
  • Theorem 2.1: DLN Geodesic Equations
  • Lemma 2.2: Characterization of Straight Lines on $\mathcal{M}$
  • Theorem 2.3: Explicit Formulas for DLN Geodesics
  • Remark 2.4
  • Remark 2.5
  • Definition 3.1: Geodesic
  • Definition 3.2: Geometric Mean
  • Lemma 3.3: Geometric Mean Properties
  • Theorem 3.4: Bures-Wasserstein Geodesics
  • ...and 13 more