Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Local properties of neural networks through the lens of layer-wise Hessians

Maxim Bolshim, Alexander Kugaevskikh

TL;DR

This paper introduces local layer-wise Hessians as a principled tool to study the geometry of neural network parameter spaces. By defining S_i(θ_i) for each functional block and computing LH_i = ∇^2_{θ_i} S_i(θ_i), the authors connect layer-wise curvature to expressivity and generalization. Through 111 experiments on 37 datasets, they show that LH_i spectra reveal architecture-specific patterns, correlate with generalization, and exhibit phase-transition-like behavior between under- and overparameterized regimes. The work provides practical guidelines for architecture optimization and motivates future integration of local geometric analysis into architecture search and training stability methods, with potential extensions to diverse architectures and geometric frameworks.

Abstract

We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matrix of second derivatives of a scalar function with respect to the parameters of that layer. This concept provides a formal tool for characterizing the local geometry of the parameter space. We show that the spectral properties of local Hessians, such as the distribution of eigenvalues, reveal quantitative patterns associated with overfitting, underparameterization, and expressivity in neural network architectures. We conduct an extensive empirical study involving 111 experiments across 37 datasets. The results demonstrate consistent structural regularities in the evolution of local Hessians during training and highlight correlations between their spectra and generalization performance. These findings establish a foundation for using local geometric analysis to guide the diagnosis and design of deep neural networks. The proposed framework connects optimization geometry with functional behavior and offers practical insight for improving network architectures and training stability.

Local properties of neural networks through the lens of layer-wise Hessians

TL;DR

This paper introduces local layer-wise Hessians as a principled tool to study the geometry of neural network parameter spaces. By defining S_i(θ_i) for each functional block and computing LH_i = ∇^2_{θ_i} S_i(θ_i), the authors connect layer-wise curvature to expressivity and generalization. Through 111 experiments on 37 datasets, they show that LH_i spectra reveal architecture-specific patterns, correlate with generalization, and exhibit phase-transition-like behavior between under- and overparameterized regimes. The work provides practical guidelines for architecture optimization and motivates future integration of local geometric analysis into architecture search and training stability methods, with potential extensions to diverse architectures and geometric frameworks.

Abstract

We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matrix of second derivatives of a scalar function with respect to the parameters of that layer. This concept provides a formal tool for characterizing the local geometry of the parameter space. We show that the spectral properties of local Hessians, such as the distribution of eigenvalues, reveal quantitative patterns associated with overfitting, underparameterization, and expressivity in neural network architectures. We conduct an extensive empirical study involving 111 experiments across 37 datasets. The results demonstrate consistent structural regularities in the evolution of local Hessians during training and highlight correlations between their spectra and generalization performance. These findings establish a foundation for using local geometric analysis to guide the diagnosis and design of deep neural networks. The proposed framework connects optimization geometry with functional behavior and offers practical insight for improving network architectures and training stability.
Paper Structure (38 sections, 2 theorems, 11 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 38 sections, 2 theorems, 11 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Paper Organization
  4. Mathematical Framework of Neural Networks
  5. Definition and Structure of a Neural Network
  6. Intermediate Representations and Activation Functions
  7. Typical Implementations in Neural Networks
  8. Local Layer-wise Hessians in Neural Networks
  9. Definition of the Local Hessian
  10. Efficient Computation of Local Hessians
  11. Algorithm for Computing the Local Hessian
  12. Mathematical Implementation Details
  13. Gradient Computation
  14. Computation of Hessian Matrix Rows
  15. Spectral Analysis of the Local Hessian
  16. ...and 23 more sections

Key Result

Lemma 1

The elements of the Hessian matrix $H_i$ can be computed sequentially by rows:

Figures (8)

  • Figure 1: Visualization of local Hessian computation for a neural network. The Hessian computation region indicates where layer-wise Hessians are calculated during forward propagation.
  • Figure 2: Comparison of CCA Score statistics across architectures. Large architectures ("huge") exhibit the highest stability with standard deviation of 0.082, while small architectures ("no") show extreme variability (std=0.976).
  • Figure 3: Comparison of spectral characteristics of third-layer gradients. The "huge" architecture shows PSD values exceeding the "no" architecture by over 100 times, indicating qualitatively different gradient propagation dynamics.
  • Figure 4: Distribution of canonical X-weights across architectures. Large architectures ("huge") show more uniform distribution, while small architectures ("no") exhibit concentrated structure with dominant components.
  • Figure 5: Distribution of canonical Y-weights across architectures. Large architectures display denser, centered distribution around zero, while small architectures show significant asymmetry with extreme outliers.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Lemma 1
  • proof
  • Proposition 1