Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Neural Networks with Two Nonlinear Layers in Polynomial Time

Surbhi Goel, Adam Klivans

TL;DR

This work delivers the first assumption-free polynomial-time algorithm for learning neural networks with two nonlinear layers, via the Alphatron framework that fuses isotonic regression with kernel methods. It unifies a broad range of results by casting learning in the probabilistic concept model and showing efficient learning for real-valued, monotone combinations of kernel-approximable functions, including DNF-related classes and monotone halfspace constructions. The key contributions include a general Alphatron theorem, two-layer neural network learnability with interpretable kernel-based features, and broad generalizations to MIL and Fourier-analyticLearnability, all under minimal distributional assumptions. The results have implications for robust, noise-tolerant learning of complex architectures and for bridging Boolean learning with real-valued probabilistic concepts in polynomial time.

Abstract

We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in $n$ dimensions (hidden weight vectors also have unit norm). This is the first assumption-free, provably efficient algorithm for learning neural networks with two nonlinear layers. Our algorithm-- {\em Alphatron}-- is a simple, iterative update rule that combines isotonic regression with kernel methods. It outputs a hypothesis that yields efficient oracle access to interpretable features. It also suggests a new approach to Boolean learning problems via real-valued conditional-mean functions, sidestepping traditional hardness results from computational learning theory. Along these lines, we subsume and improve many longstanding results for PAC learning Boolean functions to the more general, real-valued setting of {\em probabilistic concepts}, a model that (unlike PAC learning) requires non-i.i.d. noise-tolerance.

Learning Neural Networks with Two Nonlinear Layers in Polynomial Time

TL;DR

This work delivers the first assumption-free polynomial-time algorithm for learning neural networks with two nonlinear layers, via the Alphatron framework that fuses isotonic regression with kernel methods. It unifies a broad range of results by casting learning in the probabilistic concept model and showing efficient learning for real-valued, monotone combinations of kernel-approximable functions, including DNF-related classes and monotone halfspace constructions. The key contributions include a general Alphatron theorem, two-layer neural network learnability with interpretable kernel-based features, and broad generalizations to MIL and Fourier-analyticLearnability, all under minimal distributional assumptions. The results have implications for robust, noise-tolerant learning of complex architectures and for bridging Boolean learning with real-valued probabilistic concepts in polynomial time.

Abstract

We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in dimensions (hidden weight vectors also have unit norm). This is the first assumption-free, provably efficient algorithm for learning neural networks with two nonlinear layers. Our algorithm-- {\em Alphatron}-- is a simple, iterative update rule that combines isotonic regression with kernel methods. It outputs a hypothesis that yields efficient oracle access to interpretable features. It also suggests a new approach to Boolean learning problems via real-valued conditional-mean functions, sidestepping traditional hardness results from computational learning theory. Along these lines, we subsume and improve many longstanding results for PAC learning Boolean functions to the more general, real-valued setting of {\em probabilistic concepts}, a model that (unlike PAC learning) requires non-i.i.d. noise-tolerance.

Paper Structure

This paper contains 29 sections, 31 theorems, 42 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Our Approach
  4. Related Work
  5. Notation
  6. The Alphatron Algorithm
  7. General Theorems Involving Alphatron
  8. Learning Neural Networks
  9. Generalizing PAC Learning to Probabilistic Concepts
  10. Generalized DNF Learning with Queries
  11. Extending the "Low-Degree" Algorithm
  12. Learning Monotone Functions of Halfspaces with a Margin as Probabilistic Concepts
  13. Multiple Instance Learning
  14. Probabilistic MIL
  15. Background
  16. ...and 14 more sections

Key Result

Theorem 1

Let $\mathcal{K}$ be a kernel function corresponding to feature map $\psi$ such that $\forall \textbf{x} \in \mathcal{X}, ||\psi(\textbf{x})|| \leq 1$. Consider samples $(\textbf{x}_\textbf{i},y_i)_{i=1}^m$ drawn iid from distribution $\mathcal{D}$ on $\mathcal{X} \times [0,1]$ such that $E[y|\textb

Theorems & Definitions (48)

  • Theorem 1
  • Definition 1: $(\epsilon, B, M)$-approximation
  • Theorem 2
  • proof
  • Definition 2: $(\epsilon, B)$-uniform approximation
  • Theorem 3
  • proof
  • Theorem 4
  • Theorem 5
  • Lemma 1: kushilevitz1993learning
  • ...and 38 more