Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agnostic Learning of General ReLU Activation Using Gradient Descent

Pranjal Awasthi, Alex Tang, Aravindan Vijayaraghavan

TL;DR

This work establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias.

Abstract

We provide a convergence analysis of gradient descent for the problem of agnostically learning a single ReLU function with moderate bias under Gaussian distributions. Unlike prior work that studies the setting of zero bias, we consider the more challenging scenario when the bias of the ReLU function is non-zero. Our main result establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias. We also provide finite sample guarantees, and these techniques generalize to a broader class of marginal distributions beyond Gaussians.

Agnostic Learning of General ReLU Activation Using Gradient Descent

TL;DR

This work establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias.

Abstract

We provide a convergence analysis of gradient descent for the problem of agnostically learning a single ReLU function with moderate bias under Gaussian distributions. Unlike prior work that studies the setting of zero bias, we consider the more challenging scenario when the bias of the ReLU function is non-zero. Our main result establishes that starting from random initialization, in a polynomial number of iterations gradient descent outputs, with high probability, a ReLU function that achieves an error that is within a constant factor of the optimal error of the best ReLU function with moderate bias. We also provide finite sample guarantees, and these techniques generalize to a broader class of marginal distributions beyond Gaussians.
Paper Structure (31 sections, 15 theorems, 103 equations)

This paper contains 31 sections, 15 theorems, 103 equations.

Table of Contents

  1. Introduction
  2. Our Results
  3. Guarantees Beyond Gaussian marginals
  4. $O(1)$-regular marginals: Assumptions about the marginals over $\widetilde{x}$
  5. Related Work
  6. Preliminaries
  7. Gradient of the Loss.
  8. Gradient Descent.
  9. Simplification.
  10. Analysis of Gradient Descent (Proof of Theorem \ref{['thm:main-informal']})
  11. Theorem \ref{['thm:main-informal']}
  12. Challenges in arguing progress.
  13. Main proof strategy
  14. Proving progress in $\lVert w_t -v \rVert$ and $F(w_t)$.
  15. Lower bounding $\langle \nabla L(w_t), w_t - v \rangle$:
  16. ...and 16 more sections

Key Result

Theorem 1.1

Let $C_1\ge 1, C_2>0, c_3>0$ be absolute constants. Let $D$ be a distribution over $(\widetilde{x},y) \in \mathbb{R}^d \times \mathbb{R}$ where the marginal over $\widetilde{x}$ is the standard Gaussian $\mathcal{N}(0,I)$. Let $H=\{w=(\tilde{w},b_w): \|\tilde{w}\| \in [1/C_1, C_1], |b_w| \leq C_2\}

Theorems & Definitions (27)

  • Theorem 1.1
  • Theorem 1.2: Generalization beyond Gaussian marginals
  • Lemma 4.0: Lower bound on the measure of the intersection
  • Lemma 4.0: Improvement from the first order term
  • proof
  • Lemma 4.0: Success if $\norm{\nabla F} \le O(\sqrt{OPT})$
  • proof
  • Lemma 4.0: Small $\|w_t - v\|$ implies small $F(w_t)$
  • proof
  • Lemma 4.0: Decrease in $\norm{w_t-v}_2$
  • ...and 17 more