Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Power of Random Features and the Limits of Distribution-Free Gradient Descent

Ari Karchmer, Eran Malach

TL;DR

This paper establishes a fundamental limit on distribution-free gradient-based learning by showing that if a differentiable parametric model can be learned distribution-free via mini-batch SGD, then, with high probability over a prior on target functions, the target can be approximated by a polynomial-sized linear combination of random features. The core mechanism connects bSGD to statistical-query learning, bounding the required random-feature dimension by a polynomial in $T p / c^2$, and introduces the average probabilistic dimension complexity to capture average-case learnability. A Random Feature lemma grounded in communication complexity, plus a boosting construction, yields a constructive method to obtain a strong random-feature predictor from weak approximators. The results imply an infinite separation between average probabilistic dimension complexity and standard dimension complexity, and provide theoretical justification for why distributional assumptions are essential for the success of gradient-based learning in neural networks, while highlighting practical guidance to incorporate priors or distributional structure in model design.

Abstract

We study the relationship between gradient-based optimization of parametric models (e.g., neural networks) and optimization of linear combinations of random features. Our main result shows that if a parametric model can be learned using mini-batch stochastic gradient descent (bSGD) without making assumptions about the data distribution, then with high probability, the target function can also be approximated using a polynomial-sized combination of random features. The size of this combination depends on the number of gradient steps and numerical precision used in the bSGD process. This finding reveals fundamental limitations of distribution-free learning in neural networks trained by gradient descent, highlighting why making assumptions about data distributions is often crucial in practice. Along the way, we also introduce a new theoretical framework called average probabilistic dimension complexity (adc), which extends the probabilistic dimension complexity developed by Kamath et al. (2020). We prove that adc has a polynomial relationship with statistical query dimension, and use this relationship to demonstrate an infinite separation between adc and standard dimension complexity.

The Power of Random Features and the Limits of Distribution-Free Gradient Descent

TL;DR

This paper establishes a fundamental limit on distribution-free gradient-based learning by showing that if a differentiable parametric model can be learned distribution-free via mini-batch SGD, then, with high probability over a prior on target functions, the target can be approximated by a polynomial-sized linear combination of random features. The core mechanism connects bSGD to statistical-query learning, bounding the required random-feature dimension by a polynomial in , and introduces the average probabilistic dimension complexity to capture average-case learnability. A Random Feature lemma grounded in communication complexity, plus a boosting construction, yields a constructive method to obtain a strong random-feature predictor from weak approximators. The results imply an infinite separation between average probabilistic dimension complexity and standard dimension complexity, and provide theoretical justification for why distributional assumptions are essential for the success of gradient-based learning in neural networks, while highlighting practical guidance to incorporate priors or distributional structure in model design.

Abstract

We study the relationship between gradient-based optimization of parametric models (e.g., neural networks) and optimization of linear combinations of random features. Our main result shows that if a parametric model can be learned using mini-batch stochastic gradient descent (bSGD) without making assumptions about the data distribution, then with high probability, the target function can also be approximated using a polynomial-sized combination of random features. The size of this combination depends on the number of gradient steps and numerical precision used in the bSGD process. This finding reveals fundamental limitations of distribution-free learning in neural networks trained by gradient descent, highlighting why making assumptions about data distributions is often crucial in practice. Along the way, we also introduce a new theoretical framework called average probabilistic dimension complexity (adc), which extends the probabilistic dimension complexity developed by Kamath et al. (2020). We prove that adc has a polynomial relationship with statistical query dimension, and use this relationship to demonstrate an infinite separation between adc and standard dimension complexity.
Paper Structure (44 sections, 18 theorems, 73 equations, 1 algorithm)

This paper contains 44 sections, 18 theorems, 73 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. This work, in a nutshell.
  3. The theoretical value of this result.
  4. Main theorem in more detail.
  5. Techniques and further contributions.
  6. Related Work
  7. Technical Overview and Theorem Statements
  8. Learning Paradigms
  9. Mini-batch stochastic gradient descent.
  10. Statistical query learning and SQ dimension.
  11. Complexity of linear learning with random features.
  12. Complexity of linear learning with random features and a prior.
  13. Main Theorem Statement
  14. A Note on Gradient Precision.
  15. Technical Tools
  16. ...and 29 more sections

Key Result

Theorem 3.3

Suppose there exists a learning algorithm $A_{\mathrm{bSGD}}$ that is a $\mathrm{bSGD}(T, c, b, p)$ method, and $\mathrm{err}(A_{\mathrm{bSGD}}, \mathcal{D}_{f, \rho}) \le 1/10$ for every source distribution $\mathcal{D}_{f, \rho} \in \mathcal{D}_\mathcal{F}$ with respect to $\ell^\rho_{\mathrm{sq}} In other words, for any prior distribution $\mu$ over $\mathcal{F}$, $\mathrm{adc}_{\epsilon, \delt

Theorems & Definitions (38)

  • Definition 3.1: Probabilistic dimension complexity
  • Definition 3.2: Average probabilistic dimension complexity
  • Theorem 3.3: Main theorem
  • Theorem 3.4: Thm 1c. of abbe2021power
  • Definition 3.5: Statistical query dimension
  • Theorem 3.6: blum1994weakly
  • Corollary 3.7: Infinite Separation between ${ adc}_{\epsilon, \delta}^\ell$ and ${ dc}^\ell$
  • Theorem 4.1
  • proof : Proof of Theorem \ref{['thm:main_theorem']}
  • Theorem 1.1: sherstov2008halfspace - Thm. 7.1
  • ...and 28 more