Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tensor Programs III: Neural Matrix Laws

Greg Yang

TL;DR

This work develops a universal, rigorous framework (Tensor Programs/NetsorT) to analyze nonlinear random matrix problems arising in neural networks. It proves an unrestricted Master Theorem, derives new proofs of semicircle and Marchenko–Pastur laws, and introduces the Free Independence Principle, which formalizes asymptotic freeness between weight matrices and diagonal activations. Leveraging FIP, the authors provide a principled method to compute the asymptotic Jacobian singular value distribution for any architecture and relate it to free convolutions, with extensions to Neural Tangent Kernel analyses under relaxed independence assumptions. Collectively, these results offer a principled, architecture-agnostic toolkit for nonlinear random matrix theory in deep learning, with direct implications for understanding training dynamics and stability of ultra-deep networks.

Abstract

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Tensor Programs III: Neural Matrix Laws

TL;DR

This work develops a universal, rigorous framework (Tensor Programs/NetsorT) to analyze nonlinear random matrix problems arising in neural networks. It proves an unrestricted Master Theorem, derives new proofs of semicircle and Marchenko–Pastur laws, and introduces the Free Independence Principle, which formalizes asymptotic freeness between weight matrices and diagonal activations. Leveraging FIP, the authors provide a principled method to compute the asymptotic Jacobian singular value distribution for any architecture and relate it to free convolutions, with extensions to Neural Tangent Kernel analyses under relaxed independence assumptions. Collectively, these results offer a principled, architecture-agnostic toolkit for nonlinear random matrix theory in deep learning, with direct implications for understanding training dynamics and stability of ultra-deep networks.

Abstract

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Paper Structure

This paper contains 114 sections, 66 theorems, 373 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Random Matrix Theory in Deep Learning
  3. New Approach to Random Matrix Theory
  4. The expansion technique of classical RMT
  5. The expansion technique is ineffective in neural network problems
  6. The Tensor Programs Framework
  7. Calculating $\operatorname*{\mathbb{E}}_v v^\top (A^\top A)^k v$ with Tensor Programs
  8. Tensor Programs in general
  9. Tensor Programs Master Theorems
  10. NetsorT: Language for Neural Computation
  11. Extensions
  12. Semicircle Law
  13. The Main Proof Idea
  14. Examples of First Few Moments
  15. Proof for General Moments
  16. ...and 99 more sections

Key Result

Theorem 2.10

Fix a NetsorT program. Suppose the initial matrices $\mathcal{W}$ and vectors $\mathcal{V}$ are sampled in the fashion of setup:netsort. Assume all nonlinearities $\phi$ used in instr:nonlin are polynomially bounded. Then for any fixed $k$ and any polynomially bounded$\phi: \mathbb{R}^k \to \mathbb{ for any collection of vectors $h^{1},\ldots,h^{k}$ in the program, where $Z^{h^{i}}$ are defined in

Figures (3)

  • Figure 1: Graphical summary of NetsorT and its Master Theorem. Vectors $v^i$ are initial vectors of the program, but $x^i$ and $h^i$ can be any vector in the program.
  • Figure 2: Illustration of $\hat{\pi}$, $x_t$, and $t_x$.
  • Figure 3: Illustration of how to define $\hat{\zeta}$.

Theorems & Definitions (154)

  • Definition 2.1
  • Example 2.3
  • Example 2.4
  • Example 2.5
  • Remark 2.6
  • Example 2.7
  • Example 2.8
  • Remark 2.9: Intuition for definition of $Z^h$
  • Theorem 2.10: NetsorT Master Theorem
  • Remark 2.11: Partial derivative
  • ...and 144 more