Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Function Forms of Simple ReLU Networks with Random Hidden Weights

Ka Long Keith Ho, Yoshinari Takeishi, Junichi Takeuchi

TL;DR

The paper addresses how wide two-layer ReLU networks with random fixed hidden weights realize specific function forms aligned with the leading eigenvectors of the Fisher information matrix (FIM). By deriving the asymptotic limits of the functionals f_v(x) = X^T v for four eigenvector groups, it shows limiting forms F_0 ∝ ||x||, F_l ∝ x_l/2, F_{γγ} involving quadratic terms, and F_{αβ} ∝ x_α x_β/||x||, with the FIM-induced inner product rendering these directions approximately orthogonal. Theoretical results are supported by simulations across varying dimensions and widths, demonstrating that gradient descent naturally prioritizes these directions in function space. The work provides a rigorous link between parameter and function spaces via the Fisher metric and suggests extensions to NTK and deeper/random-feature settings with potential impact on initialization and architecture design.

Abstract

We investigate the function space dynamics of a two-layer ReLU neural network in the infinite-width limit, highlighting the Fisher information matrix (FIM)'s role in steering learning. Extending seminal works on approximate eigendecomposition of the FIM, we derive the asymptotic behavior of basis functions ($f_v(x) = X^{\top} v $) for four groups of approximate eigenvectors, showing their convergence to distinct function forms. These functions, prioritized by gradient descent, exhibit FIM-induced inner products that approximate orthogonality in the function space, forging a novel connection between parameter and function spaces. Simulations validate the accuracy of these theoretical approximations, confirming their practical relevance. By refining the function space inner product's role, we advance the theoretical framework for ReLU networks, illuminating their optimization and expressivity. Overall, this work offers a robust foundation for understanding wide neural networks and enhances insights into scalable deep learning architectures, paving the way for improved design and analysis of neural networks.

Function Forms of Simple ReLU Networks with Random Hidden Weights

TL;DR

The paper addresses how wide two-layer ReLU networks with random fixed hidden weights realize specific function forms aligned with the leading eigenvectors of the Fisher information matrix (FIM). By deriving the asymptotic limits of the functionals f_v(x) = X^T v for four eigenvector groups, it shows limiting forms F_0 ∝ ||x||, F_l ∝ x_l/2, F_{γγ} involving quadratic terms, and F_{αβ} ∝ x_α x_β/||x||, with the FIM-induced inner product rendering these directions approximately orthogonal. Theoretical results are supported by simulations across varying dimensions and widths, demonstrating that gradient descent naturally prioritizes these directions in function space. The work provides a rigorous link between parameter and function spaces via the Fisher metric and suggests extensions to NTK and deeper/random-feature settings with potential impact on initialization and architecture design.

Abstract

We investigate the function space dynamics of a two-layer ReLU neural network in the infinite-width limit, highlighting the Fisher information matrix (FIM)'s role in steering learning. Extending seminal works on approximate eigendecomposition of the FIM, we derive the asymptotic behavior of basis functions () for four groups of approximate eigenvectors, showing their convergence to distinct function forms. These functions, prioritized by gradient descent, exhibit FIM-induced inner products that approximate orthogonality in the function space, forging a novel connection between parameter and function spaces. Simulations validate the accuracy of these theoretical approximations, confirming their practical relevance. By refining the function space inner product's role, we advance the theoretical framework for ReLU networks, illuminating their optimization and expressivity. Overall, this work offers a robust foundation for understanding wide neural networks and enhances insights into scalable deep learning architectures, paving the way for improved design and analysis of neural networks.

Paper Structure

This paper contains 21 sections, 6 theorems, 89 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations
  4. Approximate Eigenvectors of Fisher information matrix
  5. Fisher Metric and Function Space Inner Product
  6. Main Results
  7. Simulation
  8. Setup
  9. Simulation results
  10. Conclusion and Discussion
  11. Proof of Theorem \ref{['thm1']}
  12. Proof of Theorem \ref{['thm2']}
  13. Proof of Lemma \ref{['thm3']}
  14. Case 1($x_\gamma \neq 0$, $x_{-\gamma} = 0$):
  15. Case 2 ($x_\gamma = 0$, $x_{-\gamma} \neq 0$):
  16. ...and 6 more sections

Key Result

Theorem 3.1

Let $d > 2$. For each $x \in \mathbb{R}^d$, the first approximate eigenvector $v^{(0)}$ satisfies

Figures (1)

  • Figure 1: For approximate eigenvectors $v$ and their limiting functions $F$ from Theorems \ref{['thm1']} to \ref{['thm4']}, we show the values of $X^\top v$ against the theoretical values $F(x)$. The top left shows the case of $v^{(0)}$ (Group 1), top right shows the case of $v^{(l)}$ (Group 2), bottom left shows the case of $v^{(\gamma)}$ (Group 3), and the bottom right shows the case of $v^{(\alpha,\beta)}$ (Group 3). The plots shown are generated with $d = 10$ and $m = 100000$.

Theorems & Definitions (7)

  • Theorem 3.1
  • Theorem 3.2
  • Lemma 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Lemma B.1
  • proof