Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral Thresholds for Identifiability and Stability:Finite-Sample Phase Transitions in High-Dimensional Learning

William Hao-Cheng Huang

TL;DR

This work identifies a finite-sample, necessary spectral boundary for identifiability and stability in high-dimensional learning. The central result, the Fisher spectral threshold, shows that identifiability requires the bottom Fisher eigenvalue $\lambda_{\min}(\Gamma)$ to exceed a finite-sample fluctuation floor $2\Lambda^*$, yielding a sharp phase transition and enabling linear convergence when exceeded. It introduces the Constructive Fisher Floor, a practical regularizer that enforces a minimal spectral level, and extends the theory to stochastic training with smoothing, plus robustness under preconditioning. Synthetic experiments on Gaussian mixtures and logistic models verify the $d/n$ scaling and demonstrate how smoothing, regularization, and finite-direction monitoring can diagnose and enforce stability. Overall, the paper reframes classical eigenvalue conditions into a non-asymptotic spectral law, bridging statistical identifiability with learning-theoretic stability and providing actionable tools for robust high-dimensional inference.

Abstract

In high-dimensional learning, models remain stable until they collapse abruptly once the sample size falls below a critical level. This instability is not algorithm-specific but a geometric mechanism: when the weakest Fisher eigendirection falls beneath sample-level fluctuations, identifiability fails. Our Fisher Threshold Theorem formalizes this by proving that stability requires the minimal Fisher eigenvalue to exceed an explicit $O(\sqrt{d/n})$ bound. Unlike prior asymptotic or model-specific criteria, this threshold is finite-sample and necessary, marking a sharp phase transition between reliable concentration and inevitable failure. To make the principle constructive, we introduce the Fisher floor, a verifiable spectral regularization robust to smoothing and preconditioning. Synthetic experiments on Gaussian mixtures and logistic models confirm the predicted transition, consistent with $d/n$ scaling. Statistically, the threshold sharpens classical eigenvalue conditions into a non-asymptotic law; learning-theoretically, it defines a spectral sample-complexity frontier, bridging theory with diagnostics for robust high-dimensional inference.

Spectral Thresholds for Identifiability and Stability:Finite-Sample Phase Transitions in High-Dimensional Learning

TL;DR

This work identifies a finite-sample, necessary spectral boundary for identifiability and stability in high-dimensional learning. The central result, the Fisher spectral threshold, shows that identifiability requires the bottom Fisher eigenvalue to exceed a finite-sample fluctuation floor , yielding a sharp phase transition and enabling linear convergence when exceeded. It introduces the Constructive Fisher Floor, a practical regularizer that enforces a minimal spectral level, and extends the theory to stochastic training with smoothing, plus robustness under preconditioning. Synthetic experiments on Gaussian mixtures and logistic models verify the scaling and demonstrate how smoothing, regularization, and finite-direction monitoring can diagnose and enforce stability. Overall, the paper reframes classical eigenvalue conditions into a non-asymptotic spectral law, bridging statistical identifiability with learning-theoretic stability and providing actionable tools for robust high-dimensional inference.

Abstract

In high-dimensional learning, models remain stable until they collapse abruptly once the sample size falls below a critical level. This instability is not algorithm-specific but a geometric mechanism: when the weakest Fisher eigendirection falls beneath sample-level fluctuations, identifiability fails. Our Fisher Threshold Theorem formalizes this by proving that stability requires the minimal Fisher eigenvalue to exceed an explicit bound. Unlike prior asymptotic or model-specific criteria, this threshold is finite-sample and necessary, marking a sharp phase transition between reliable concentration and inevitable failure. To make the principle constructive, we introduce the Fisher floor, a verifiable spectral regularization robust to smoothing and preconditioning. Synthetic experiments on Gaussian mixtures and logistic models confirm the predicted transition, consistent with scaling. Statistically, the threshold sharpens classical eigenvalue conditions into a non-asymptotic law; learning-theoretically, it defines a spectral sample-complexity frontier, bridging theory with diagnostics for robust high-dimensional inference.

Paper Structure

This paper contains 51 sections, 11 theorems, 69 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Statistical Identifiability.
  4. Spectral Phase Transitions.
  5. Algorithmic Stability and Generalization.
  6. Positioning.
  7. Preliminaries
  8. Setup and Notation.
  9. Assumptions.
  10. Frequently Used Consequences.
  11. Role in the Paper.
  12. Main Theoretical Results
  13. Fisher Spectral Threshold (Theorem 1)
  14. Intuition.
  15. Proof sketch.
  16. ...and 36 more sections

Key Result

Theorem 4.1

Assume (A1)–(A3) on $B_r(\theta^\ast)$. With probability at least $1-\delta$, if $\lambda_{\min}(\Gamma)\ge 2\Lambda^\ast$, then $L$ satisfies the PL inequality yielding linear convergence of gradient descent karimi2016linearpolyak1963gradient. Conversely, if $\lambda_{\min}(\Gamma)\le \tfrac{1}{2}\Lambda^\ast$, then indistinguishable local alternatives exist (via Le Cam) lecam2000asymptotics, so

Figures (4)

  • Figure 1: Experiment A: Sample size and phase transition. Accuracy stabilizes exactly when $\lambda_{\min}$ crosses $2\Lambda^\ast$, confirming Theorem \ref{['thm:threshold']}.
  • Figure 2: Experiments A (PL) and B: PL geometry and two-point indistinguishability. Left: Gradient–loss slope exceeds $\mu_{\min}$, verifying PL geometry. Middle/right: LRT error matches Le Cam’s $1/4$ bound below threshold, and decreases with $\rho$ above threshold.
  • Figure 3: Experiments C and D: Smoothing interventions and Fisher-floor regularization. Left: Smoothing lifts $\lambda_{\min}$ above $2\Lambda^\ast$ at $\sigma^\ast$. Middle/right: Fisher floor keeps $\lambda_{\min}\ge\tau$ and scales linearly with $\tau$, enforcing stability.
  • Figure 4: Experiment E: Finite-direction monitoring and angle penalty. Left: Tracked $\phi_K$ converges to $\lambda_{\min}$. Right: Error gap decays as predicted by $\Delta_B \sin^2\vartheta$.

Theorems & Definitions (18)

  • Theorem 4.1: Finite-sample spectral threshold (tight PL constant)
  • Corollary 4.2: Stochastic extension via smoothing and robust Fisher concentration
  • Theorem 4.3: Constructive Fisher floor
  • Corollary 4.4: Finite-direction monitoring
  • Proposition 4.5: Robustness under preconditioning
  • Lemma A.0: Frequently used facts under (A1)--(A3)
  • proof
  • proof
  • Theorem A.1: Finite-sample spectral phase transition (tight PL constant)
  • proof
  • ...and 8 more