Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dimension-independent learning rates for high-dimensional classification problems

Andres Felipe Lerma-Pineda, Philipp Petersen, Simon Frieder, Thomas Lukasiewicz

TL;DR

It is modified to show that every $RBV^2 function can be approximated by a neural network with bounded weights, and it is proved the existence of a neural network with bounded weights approximating a classification function.

Abstract

We study the problem of approximating and estimating classification functions that have their decision boundary in the $RBV^2$ space. Functions of $RBV^2$ type arise naturally as solutions of regularized neural network learning problems and neural networks can approximate these functions without the curse of dimensionality. We modify existing results to show that every $RBV^2$ function can be approximated by a neural network with bounded weights. Thereafter, we prove the existence of a neural network with bounded weights approximating a classification function. And we leverage these bounds to quantify the estimation rates. Finally, we present a numerical study that analyzes the effect of different regularity conditions on the decision boundaries.

Dimension-independent learning rates for high-dimensional classification problems

TL;DR

It is modified to show that every $RBV^2 function can be approximated by a neural network with bounded weights, and it is proved the existence of a neural network with bounded weights approximating a classification function.

Abstract

We study the problem of approximating and estimating classification functions that have their decision boundary in the space. Functions of type arise naturally as solutions of regularized neural network learning problems and neural networks can approximate these functions without the curse of dimensionality. We modify existing results to show that every function can be approximated by a neural network with bounded weights. Thereafter, we prove the existence of a neural network with bounded weights approximating a classification function. And we leverage these bounds to quantify the estimation rates. Finally, we present a numerical study that analyzes the effect of different regularity conditions on the decision boundaries.
Paper Structure (13 sections, 11 theorems, 98 equations, 1 figure)

This paper contains 13 sections, 11 theorems, 98 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Our contribution
  3. Related work
  4. Structure of the paper
  5. Neural Networks
  6. Approximation of functions
  7. The set of $RBV^2$ functions
  8. Approximation of $RBV^2$ functions by neural networks
  9. Upper bounds on learning horizon functions
  10. Estimation bounds
  11. Numerical Experiments
  12. Experiment set-up
  13. Evaluation and interpretations of the results

Key Result

Proposition 1.1

Let $d \in \mathbb{N}$, $d \geq 3$ and let $f \in RBV^2(B_1^d)$. Then, for every $N \in \mathbb{N}$, there is a shallow $NN$$f_N$ with $N$ neurons in the hidden layer and with weights bounded by a constant $C>0$ that depends linearly on $\Vert f \Vert_{RBV^2}$ such that

Figures (1)

  • Figure 1: Plot of the mean relative test error for the sets of case $d=2$ (left), $d=3$ (middle) and $d=4$ (right). The number of samples varies along the horizontal axis and the mean relative test error is shown on the vertical axis.

Theorems & Definitions (33)

  • Proposition 1.1
  • Theorem 1.2
  • Definition 2.1
  • Definition 2.2
  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Definition 3.4
  • Definition 3.5
  • Definition 3.6
  • ...and 23 more