Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Yaoqing Yang

TL;DR

This work shows that Hessian spectral bifurcation in deep networks can arise from architecture alone, even when data covariances are balanced. Through a rigorous analysis of depth-$L$ deep linear networks with whitened inputs, the authors decompose the Hessian via Gauss-Newton and prove a two-cluster spectrum whose dominant-to-bulk gap grows linearly with depth: $\lambda_{\text{dom}}/\lambda_{\text{bulk}} = \Theta(L)$. A Uniform Spectral Initialization (USI) further yields exactly two nonzero eigenvalues with ratio $L$, highlighting a clean mechanism for depth-induced conditioning. Simulations on whitened data corroborate the theory, underscoring that optimization strategies should account for both architecture and data characteristics.

Abstract

The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.

Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

TL;DR

This work shows that Hessian spectral bifurcation in deep networks can arise from architecture alone, even when data covariances are balanced. Through a rigorous analysis of depth- deep linear networks with whitened inputs, the authors decompose the Hessian via Gauss-Newton and prove a two-cluster spectrum whose dominant-to-bulk gap grows linearly with depth: . A Uniform Spectral Initialization (USI) further yields exactly two nonzero eigenvalues with ratio , highlighting a clean mechanism for depth-induced conditioning. Simulations on whitened data corroborate the theory, underscoring that optimization strategies should account for both architecture and data characteristics.

Abstract

The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.
Paper Structure (18 sections, 17 theorems, 162 equations, 16 figures)

This paper contains 18 sections, 17 theorems, 162 equations, 16 figures.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Hessian Bifurcation Theory
  4. Theorem Assumptions
  5. Main Result
  6. Simulation
  7. Conclusion
  8. Proof of Main Theorem
  9. Notation and Problem Setup
  10. Notation and Matrix Operations.
  11. Deep Linear Network Architecture.
  12. Data Distribution and Loss Function.
  13. Gauss-Newton Decomposition.
  14. Balanced Initialization.
  15. Gradient Descent Dynamics.
  16. ...and 3 more sections

Key Result

Lemma 3.4

Under Assumptions assumption 12 and assumption 4, all weight matrices $W^k_t$ for $k = 1, \ldots, L$ share a same spectral structure $\Sigma_t^{1/L} = \text{diag}(\lambda_{1,t}, \lambda_{2,t}, \ldots, \lambda_{d_*,t})$ at any time $t \geq 0$. Specifically: where $U \in \mathbb{R}^{d_L \times d_*}$ and $V \in \mathbb{R}^{d_0 \times d_*}$ are the left and right singular vector matrices from the bal

Figures (16)

  • Figure 1: Eigenvalue distribution of the Hessian for a deep linear network training on whitened data. Despite the data cross covariance matrix $\Sigma_{yx}$ being balanced, the Hessian spectrum splits into a dominant cluster and a bulk cluster.
  • Figure 2: Evolution of Hessian eigenvalues and training loss for deep linear networks with increasing depth $L$. Left: Hessian eigenvalue trajectories; right: training loss. Under whitened data ($\Sigma_{xx} \simeq I_{d_\ast}$), the non-zero spectrum splits into a dominant cluster ($r^2$) and a bulk cluster $(d_\ast + d_L - 2r)r$. Dominant eigenvalues grow approximately linearly with $L$, consistent with Theorem \ref{['theorem main result']}.
  • Figure 3: Eigenvalue evolution. The curves are color-coded by subspace: purple for the dominant space, orange for the bulk space, and green for the near-zero space. The dominant space has a dimension of $\text{rank}^2$, while the combined dimension of the dominant and bulk spaces equals the product of the input and output dimensions. The final eigenvalues of the dominant space converge to $L$ times those of the bulk space. The panels correspond to $L=3$ (left), $L=4$ (middle), and $L=5$ (right).
  • Figure 4: Eigenvalue evolution. The curves are color-coded by subspace: purple for the dominant space, orange for the bulk space, and green for the near-zero space. The dominant space has a dimension of $\text{rank}^2$, while the combined dimension of the dominant and bulk spaces equals the product of the input and output dimensions. The final eigenvalues of the dominant space converge to $L$ times those of the bulk space. The panels correspond to $L=3$ (left), $L=4$ (middle), and $L=5$ (right).
  • Figure 5: Eigenvalue evolution. The curves are color-coded by subspace: purple for the dominant space, orange for the bulk space, and green for the near-zero space. The dominant space has a dimension of $\text{rank}^2$, while the combined dimension of the dominant and bulk spaces equals the product of the input and output dimensions. The final eigenvalues of the dominant space converge to $L$ times those of the bulk space. The panels correspond to $L=3$ (left), $L=4$ (middle), and $L=5$ (right).
  • ...and 11 more figures

Theorems & Definitions (31)

  • Lemma 3.4: Shared Spectral Structure
  • Theorem 3.5: Hessian Bifurcation
  • Corollary 3.6: Hessian Bifurcation with USI
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • Lemma A.3
  • proof
  • Lemma A.4
  • ...and 21 more