Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-dimensional classification problems with Barron regular boundaries under margin conditions

Jonathan García, Philipp Petersen

TL;DR

This work addresses high-dimensional binary classification where the decision boundary is Barron-regular and the data satisfy a margin condition. By leveraging Fourier-analytic Barron spaces and a hinge-loss framework, the authors prove that a three-hidden-layer ReLU network can approximate the indicator of the target region at a rate $\mu(\{\mathbb{1}_{\Omega} \neq \Phi\}) \lesssim N^{-\gamma/2}$ under margin exponent $\gamma$ and tube-compatibility, and they establish fast learning rates of $O(n^{-\gamma/(2+\gamma)}(1+\log n))$ that approach $n^{-1}$ as $\gamma$ grows. The results imply that high-dimensional discontinuous classification can be learned at rates comparable to low-dimensional smooth functions when margin conditions are strong. Numerical experiments across dimensions $d\in\{3,50,784\}$ (including MNIST-scale data) corroborate the theory, demonstrating practical performance gains with increasing margin. Overall, the paper provides a framework to overcome the curse of dimensionality under margin assumptions by tying Barron-regular boundaries to efficient ReLU-NN approximations and fast learning rates.

Abstract

We prove that a classifier with a Barron-regular decision boundary can be approximated with a rate of high polynomial degree by ReLU neural networks with three hidden layers when a margin condition is assumed. In particular, for strong margin conditions, high-dimensional discontinuous classifiers can be approximated with a rate that is typically only achievable when approximating a low-dimensional smooth function. We demonstrate how these expression rate bounds imply fast-rate learning bounds that are close to $n^{-1}$ where $n$ is the number of samples. In addition, we carry out comprehensive numerical experimentation on binary classification problems with various margins. We study three different dimensions, with the highest dimensional problem corresponding to images from the MNIST data set.

High-dimensional classification problems with Barron regular boundaries under margin conditions

TL;DR

This work addresses high-dimensional binary classification where the decision boundary is Barron-regular and the data satisfy a margin condition. By leveraging Fourier-analytic Barron spaces and a hinge-loss framework, the authors prove that a three-hidden-layer ReLU network can approximate the indicator of the target region at a rate under margin exponent and tube-compatibility, and they establish fast learning rates of that approach as grows. The results imply that high-dimensional discontinuous classification can be learned at rates comparable to low-dimensional smooth functions when margin conditions are strong. Numerical experiments across dimensions (including MNIST-scale data) corroborate the theory, demonstrating practical performance gains with increasing margin. Overall, the paper provides a framework to overcome the curse of dimensionality under margin assumptions by tying Barron-regular boundaries to efficient ReLU-NN approximations and fast learning rates.

Abstract

We prove that a classifier with a Barron-regular decision boundary can be approximated with a rate of high polynomial degree by ReLU neural networks with three hidden layers when a margin condition is assumed. In particular, for strong margin conditions, high-dimensional discontinuous classifiers can be approximated with a rate that is typically only achievable when approximating a low-dimensional smooth function. We demonstrate how these expression rate bounds imply fast-rate learning bounds that are close to where is the number of samples. In addition, we carry out comprehensive numerical experimentation on binary classification problems with various margins. We study three different dimensions, with the highest dimensional problem corresponding to images from the MNIST data set.

Paper Structure

This paper contains 14 sections, 4 theorems, 88 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Neural networks
  4. Loss function and the hinge loss
  5. Covering entropy
  6. Fourier-analytic Barron space
  7. Barron regular boundaries
  8. Main results
  9. Numerical experiments
  10. Sample generation
  11. NN training
  12. Appendix
  13. Proof of Theorem \ref{['firstaproxtheo']}
  14. Proof of Theorem \ref{['secondproxtheo']}

Key Result

Lemma 1

Fixing $\mathcal{X}:=\mathcal{NN}_{*}(d,N,W,B)$, $K:=[0,1]^{d}$ and $\operatorname{dist}:=\lVert\cdot\rVert_{\infty}$, we have for $\delta\in (0,1], d,N,W\in \mathbb{N}$ and $B>0$.

Figures (8)

  • Figure 1: Examples test for $d=3$.
  • Figure 2: Some examples after smote for $d=784$.
  • Figure 3: Train sample probability applying $\Phi_{0}$ and decision boundary for $d=784$.
  • Figure 4: Final train sample after SMOTE and location of the possible $\partial\Omega^{784}$.
  • Figure 5: Example of margin condition applied for $d=784$. The second row contains the points of the first row but in ascending order.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Lemma 1
  • Theorem 2
  • Theorem 3
  • Remark 4
  • Lemma 5
  • proof