Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks

Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz

TL;DR

This work advances the theoretical understanding of SGD learning for non-linear, regular neural networks by focusing on depth-2 architectures trained in the mean-field regime on high-dimensional, sparse latent functions. It introduces the merged-staircase property (MSP) as a necessary and nearly sufficient condition for strong SGD-learnability with $O(d)$ samples, and couples this with a dimension-free mean-field dynamic to characterize learning in latent subspaces. The results show that MSP enables efficient SGD learning while linear methods fail to adapt to latent sparsity, with generic MSP functions being learnable almost surely and degenerate MSPs forming a measure-zero set. Together, these findings provide a tight, fine-grained characterization of when regular SGD on regular networks can efficiently learn sparse functions, and they establish a concrete separation from fixed-feature/ kernel-based approaches with implications for understanding neural-network generalization in high dimensions.

Abstract

It is currently known how to characterize functions that neural networks can learn with SGD for two extremal parameterizations: neural networks in the linear regime, and neural networks with no structural constraints. However, for the main parametrization of interest (non-linear but regular networks) no tight characterization has yet been achieved, despite significant developments. We take a step in this direction by considering depth-2 neural networks trained by SGD in the mean-field regime. We consider functions on binary inputs that depend on a latent low-dimensional subspace (i.e., small number of coordinates). This regime is of interest since it is poorly understood how neural networks routinely tackle high-dimensional datasets and adapt to latent low-dimensional structure without suffering from the curse of dimensionality. Accordingly, we study SGD-learnability with $O(d)$ sample complexity in a large ambient dimension $d$. Our main results characterize a hierarchical property, the "merged-staircase property", that is both necessary and nearly sufficient for learning in this setting. We further show that non-linear training is necessary: for this class of functions, linear methods on any feature map (e.g., the NTK) are not capable of learning efficiently. The key tools are a new "dimension-free" dynamics approximation result that applies to functions defined on a latent space of low-dimension, a proof of global convergence based on polynomial identity testing, and an improvement of lower bounds against linear methods for non-almost orthogonal functions.

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks

TL;DR

This work advances the theoretical understanding of SGD learning for non-linear, regular neural networks by focusing on depth-2 architectures trained in the mean-field regime on high-dimensional, sparse latent functions. It introduces the merged-staircase property (MSP) as a necessary and nearly sufficient condition for strong SGD-learnability with samples, and couples this with a dimension-free mean-field dynamic to characterize learning in latent subspaces. The results show that MSP enables efficient SGD learning while linear methods fail to adapt to latent sparsity, with generic MSP functions being learnable almost surely and degenerate MSPs forming a measure-zero set. Together, these findings provide a tight, fine-grained characterization of when regular SGD on regular networks can efficiently learn sparse functions, and they establish a concrete separation from fixed-feature/ kernel-based approaches with implications for understanding neural-network generalization in high dimensions.

Abstract

It is currently known how to characterize functions that neural networks can learn with SGD for two extremal parameterizations: neural networks in the linear regime, and neural networks with no structural constraints. However, for the main parametrization of interest (non-linear but regular networks) no tight characterization has yet been achieved, despite significant developments. We take a step in this direction by considering depth-2 neural networks trained by SGD in the mean-field regime. We consider functions on binary inputs that depend on a latent low-dimensional subspace (i.e., small number of coordinates). This regime is of interest since it is poorly understood how neural networks routinely tackle high-dimensional datasets and adapt to latent low-dimensional structure without suffering from the curse of dimensionality. Accordingly, we study SGD-learnability with sample complexity in a large ambient dimension . Our main results characterize a hierarchical property, the "merged-staircase property", that is both necessary and nearly sufficient for learning in this setting. We further show that non-linear training is necessary: for this class of functions, linear methods on any feature map (e.g., the NTK) are not capable of learning efficiently. The key tools are a new "dimension-free" dynamics approximation result that applies to functions defined on a latent space of low-dimension, a proof of global convergence based on polynomial identity testing, and an improvement of lower bounds against linear methods for non-almost orthogonal functions.
Paper Structure (83 sections, 59 theorems, 379 equations, 6 figures)

This paper contains 83 sections, 59 theorems, 379 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Summary of main results
  3. Further related literature
  4. Strong SGD-learnability in $O(d)$-scaling
  5. Continuous dimension-free dynamics and necessary condition
  6. Sufficient conditions for strong SGD-learnability
  7. Degenerate cases:
  8. Generic MSP functions are learnable:
  9. Vanilla staircase, learnable without genericity:
  10. Proof ideas
  11. Separation with linear methods
  12. Conclusion and future directions
  13. Additional numerical simulations
  14. Proofs for continuous mean-field and dimension-free dynamics
  15. Justification for the dimension-free dynamics equations
  16. ...and 68 more sections

Key Result

Theorem 5

Assume conditions ${\rm A}0$-${\rm A}2$,${\rm A}3'$ hold, and let $T \geq 1$. There exist constants $K_0$ and $K_1$ depending only on the constants in ${\rm A} 0$-${\rm A} 2$,${\rm A}3'$ (in particular, independent of $d,P,T$), such that for any $b\leq d$, $N\leq e^d$, $\eta \leq e^{-K_0 T^3} b/(d+\ for all $k \in [T/\eta] \cap {\mathbb N}$, with probability at least $1 - 1/N$.

Figures (6)

  • Figure 1: Comparison between \ref{['eq:bSGD']} and \ref{['eq:DF-PDE']} dynamics for $h_* ({\boldsymbol z}) = z_1 + z_1 z_2 + z_1 z_2 z_3 + z_1 z_2 z_3 z_4$. Left: Test error. Right: Fourier coefficients of $\hat{f}_{{\sf NN}} ({\boldsymbol x} ; {\boldsymbol \Theta}^{t/\eta})$ and $\hat{f}_{{\sf NN}} ({\boldsymbol z} ; \overline{\rho}_t)$. The dashed-dotted black lines correspond to \ref{['eq:DF-PDE']} and the continuous colored line to \ref{['eq:bSGD']}. The test errors and Fourier coefficients are evaluated with $m=300$ test samples and for \ref{['eq:bSGD']}, we report the average and $95\%$ confidence interval over 10 experiments.
  • Figure 2: Evolution of the Fourier coefficients during the \ref{['eq:DF-PDE']} dynamics for $4$ MSP functions.
  • Figure 3: Evolution of Fourier coefficients during the \ref{['eq:DF-PDE']} dynamics for the degenerate MSP $h_4 ({\boldsymbol z}) = z_1 + z_2 + z_3 + z_1z_2z_3$ (left) and perturbed MSP function $\Tilde{h}_4 ({\boldsymbol z}) = z_1 + 0.99 z_2 + 1.01 z_3 + z_1z_2z_3$ (right).
  • Figure 4: Fourier coefficients of one-pass \ref{['eq:bSGD']} and \ref{['eq:DF-PDE']} solutions throughout the dynamics for $h_* ({\boldsymbol z}) = z_1 + z_1 z_2 z_3 + z_1 z_2 z_3 z_4$ (left) and $h_* ({\boldsymbol z}) = z_1 + z_1 z_2 z_3 z_4$ (right).
  • Figure 5: The bump function $\sigma_{\mathrm{bump}}(u;\alpha,\beta,\gamma)$ with $\alpha = 2,\beta = 8,\gamma = 1$.
  • ...and 1 more figures

Theorems & Definitions (178)

  • Definition 1
  • Definition 2: Merged-Staircase Property
  • Definition 3: SGD-learnability in $O(d^\alpha)$-scaling
  • Definition 4: Strong SGD-learnability in $O(d)$-scaling
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Definition 8
  • Theorem 9
  • Theorem 10
  • ...and 168 more