Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective

Shokichi Takakura, Taiji Suzuki

TL;DR

The work addresses the limitation of lazy kernel regimes (e.g., NTK) by studying feature learning in two-layer networks through a mean-field lens under a two-timescale limit, linking training dynamics to kernel learning.By deriving a convex limiting functional $G(\mu)$ whose optimal $a_\mu$ induces the kernel $k_\mu$, the authors show global convergence of the mean-field Langevin dynamics and quantify time and particle discretization errors.The analysis demonstrates that networks can learn a data-dependent kernel that aligns with the target function, achieving better sample complexity than any fixed kernel for a union of Barron-type RKHSs, and provides a generalization bound via Barron space considerations.To control overfitting and noise, the paper introduces a label-noise procedure that regularizes the kernel through degrees of freedom, with provable convergence to the global optimum and empirical improvements in generalization.Overall, the work connects mean-field neural networks to kernel-learning theory, showing feature learning yields adaptive kernels and practical mechanisms to regulate complexity in high-dimensional settings.

Abstract

In this paper, we study the feature learning ability of two-layer neural networks in the mean-field regime through the lens of kernel methods. To focus on the dynamics of the kernel induced by the first layer, we utilize a two-timescale limit, where the second layer moves much faster than the first layer. In this limit, the learning problem is reduced to the minimization problem over the intrinsic kernel. Then, we show the global convergence of the mean-field Langevin dynamics and derive time and particle discretization error. We also demonstrate that two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than any kernel methods, and neural networks acquire data-dependent kernel which aligns with the target function. In addition, we develop a label noise procedure, which converges to the global optimum and show that the degrees of freedom appears as an implicit regularization.

Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective

TL;DR

The work addresses the limitation of lazy kernel regimes (e.g., NTK) by studying feature learning in two-layer networks through a mean-field lens under a two-timescale limit, linking training dynamics to kernel learning.By deriving a convex limiting functional $G(\mu)$ whose optimal $a_\mu$ induces the kernel $k_\mu$, the authors show global convergence of the mean-field Langevin dynamics and quantify time and particle discretization errors.The analysis demonstrates that networks can learn a data-dependent kernel that aligns with the target function, achieving better sample complexity than any fixed kernel for a union of Barron-type RKHSs, and provides a generalization bound via Barron space considerations.To control overfitting and noise, the paper introduces a label-noise procedure that regularizes the kernel through degrees of freedom, with provable convergence to the global optimum and empirical improvements in generalization.Overall, the work connects mean-field neural networks to kernel-learning theory, showing feature learning yields adaptive kernels and practical mechanisms to regulate complexity in high-dimensional settings.

Abstract

In this paper, we study the feature learning ability of two-layer neural networks in the mean-field regime through the lens of kernel methods. To focus on the dynamics of the kernel induced by the first layer, we utilize a two-timescale limit, where the second layer moves much faster than the first layer. In this limit, the learning problem is reduced to the minimization problem over the intrinsic kernel. Then, we show the global convergence of the mean-field Langevin dynamics and derive time and particle discretization error. We also demonstrate that two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than any kernel methods, and neural networks acquire data-dependent kernel which aligns with the target function. In addition, we develop a label noise procedure, which converges to the global optimum and show that the degrees of freedom appears as an implicit regularization.
Paper Structure (34 sections, 23 theorems, 166 equations, 2 figures)

This paper contains 34 sections, 23 theorems, 166 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Relation between Neural Networks and Kernel Methods
  4. Mean-field Analysis
  5. Two-timescale Limit
  6. Feature Learning in Two-layer Neural Networks
  7. Implicit Bias of Label Noise
  8. Notations
  9. Problem Settings
  10. Mean-field and Two-timescale Limit in Two-layer Neural Networks
  11. Kernel Induced by the First Layer
  12. Mean-field Langevin Dynamics
  13. Convergence Analysis
  14. Generalization Error for Barron Spaces and Superiority to Kernel Methods
  15. Properties of the Kernel Induced by the First Layer
  16. ...and 19 more sections

Key Result

Theorem 3.1

Assume that the losses $\qty{l_i}_{i=1}^T$ are convex. Then, the limiting functional $G(\mu)$ is convex. That is, it holds that for any $\mu_1, \mu_2 \in \mathcal{P}$. In addition, the first variation of $G(\mu)$ is given by

Figures (2)

  • Figure 1: Evolution of the kernel alignment and the degrees of freedom of neural network optimized by the MFLD
  • Figure 2: Evolution of the degrees of freedom and the test error of the label noise procedure

Theorems & Definitions (41)

  • Theorem 3.1
  • Definition 3.2
  • Lemma 3.5
  • Remark 3.6
  • Theorem 3.7
  • Proposition 3.8
  • Definition 4.2: KL-restricted Barron space
  • Lemma 4.3
  • Theorem 4.4
  • Theorem 4.5
  • ...and 31 more