Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrization

Giovanni Luca Marchetti, Erin Connelly, Paul Breiding, Kathlén Kohn

TL;DR

The paper addresses the problem of counting and characterizing critical points for degenerate quadratic objectives on algebraic varieties, a situation arising in overparametrized learning. It introduces a reduction to a nondegenerate problem via the orthogonal projection onto the kernel’s orthogonal complement $K^{\perp}$ and analyzes the ramification and branch loci to classify critical points and their counts using generalized Euclidean distance degrees and polar degrees. A two-regime theory is developed: (i) mild degeneracy ($k < n-d$) reduces to the EDD of the projected variety, and (ii) strong degeneracy ($k\ge n-d$) yields zero-loss solutions plus ramification-branch contributions, counted by the EDD of the branch locus. The framework is illustrated with linear neural networks and a self-attention mechanism, highlighting both generic and non-generic (degenerate) behaviors and connecting algebraic geometry with practical ML scenarios.

Abstract

We study the critical points over an algebraic variety of an optimization problem defined by a quadratic objective that is degenerate. This scenario arises in machine learning when the dataset size is small with respect to the model, and is typically referred to as overparametrization. Our main result relates the degenerate optimization problem to a nondegenerate one via a projection. In the highly-degenerate regime, we find that a central role is played by the ramification locus of the projection. Additionally, we provide tools for counting the number of critical points over projective varieties, and discuss specific cases arising from deep learning. Our work bridges tools from algebraic geometry with ideas from machine learning, and it extends the line of literature around the Euclidean distance degree to the degenerate setting.

Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrization

TL;DR

The paper addresses the problem of counting and characterizing critical points for degenerate quadratic objectives on algebraic varieties, a situation arising in overparametrized learning. It introduces a reduction to a nondegenerate problem via the orthogonal projection onto the kernel’s orthogonal complement and analyzes the ramification and branch loci to classify critical points and their counts using generalized Euclidean distance degrees and polar degrees. A two-regime theory is developed: (i) mild degeneracy () reduces to the EDD of the projected variety, and (ii) strong degeneracy () yields zero-loss solutions plus ramification-branch contributions, counted by the EDD of the branch locus. The framework is illustrated with linear neural networks and a self-attention mechanism, highlighting both generic and non-generic (degenerate) behaviors and connecting algebraic geometry with practical ML scenarios.

Abstract

We study the critical points over an algebraic variety of an optimization problem defined by a quadratic objective that is degenerate. This scenario arises in machine learning when the dataset size is small with respect to the model, and is typically referred to as overparametrization. Our main result relates the degenerate optimization problem to a nondegenerate one via a projection. In the highly-degenerate regime, we find that a central role is played by the ramification locus of the projection. Additionally, we provide tools for counting the number of critical points over projective varieties, and discuss specific cases arising from deep learning. Our work bridges tools from algebraic geometry with ideas from machine learning, and it extends the line of literature around the Euclidean distance degree to the degenerate setting.
Paper Structure (19 sections, 15 theorems, 82 equations, 3 figures)

This paper contains 19 sections, 15 theorems, 82 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Overview of Main Results
  3. Background
  4. Algebraic sets
  5. Quadratic optimization
  6. Machine Learning Perspective
  7. General Varieties
  8. Mildly degenerate quadrics ($k+d < n$)
  9. Significantly degenerate quadrics ($k+d \geq n$)
  10. Proof of Lemma \ref{['lem:piBirationalOnRam']}
  11. Counting over Projective Varieties
  12. Case Studies
  13. Linear Neural Networks
  14. A Self-Attention Mechanism
  15. The neuromanifold and its ambient space.
  16. ...and 4 more sections

Key Result

Theorem 1.2

For almost allMore precisely, almost all refers to generic objects. quadratic forms $Q$ with a $k$-dimensional kernel and almost all $u \in \mathbb R^n$, the critical points $x$ of $Q(x-u)$ satisfy the following: Case 1:$k < n -d$ (i.e., the quadric is mildly degenerate, its kernel has dimension les

Figures (3)

  • Figure 1: A slice of the neuromanifold of lightning self-attention mechanisms (adapted from henry2024geometry).
  • Figure 2: Illustration of different scenarios, where $X$ is either a curve (a) or a surface (b), (c) or (d) in $\mathbb R^3$.
  • Figure 3: Critical points (yellow) and normal lines (dashed) on the circle w.r.t. two different quadrics $Q$.

Theorems & Definitions (37)

  • Example 1.1
  • Theorem 1.2: Main Result, informal version
  • Example 2.1
  • Example 2.2
  • Example 2.3
  • Example 2.4
  • Example 2.5
  • Proposition 2.6
  • proof
  • Example 2.7
  • ...and 27 more