Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal neural network approximation of smooth compositional functions on sets with low intrinsic dimension

Thomas Nagler, Sophie Langer

TL;DR

This work analyzes deep ReLU networks under structural assumptions that mitigate dimensionality challenges. It proves minimax-optimal uniform approximation rates for $s$-Hölder smooth functions on sets with low Minkowski dimension via networks with flexible width and depth, and introduces a novel memorization result enabling efficient point fitting in dense architectures. It then defines a compositional model where each component acts on a low-dimensional domain and shows that deep nets can approximate such functions with rates governed by the most difficult component, unifying structure and dimension. The results yield improved convergence rates for empirical risk minimization in nonparametric regression, adapting to smoothness, compositional structure, and intrinsic dimensionality, and providing a potential theoretical basis for the practical success of deep networks on high-dimensional tasks with latent low-dimensional structure.

Abstract

We study approximation and statistical learning properties of deep ReLU networks under structural assumptions that mitigate the curse of dimensionality. We prove minimax-optimal uniform approximation rates for $s$-Hölder smooth functions defined on sets with low Minkowski dimension using fully connected networks with flexible width and depth, improving existing results by logarithmic factors even in classical full-dimensional settings. A key technical ingredient is a new memorization result for deep ReLU networks that enables efficient point fitting with dense architectures. We further introduce a class of compositional models in which each component function is smooth and acts on a domain of low intrinsic dimension. This framework unifies two common assumptions in the statistical learning literature, structural constraints on the target function and low dimensionality of the covariates, within a single model. We show that deep networks can approximate such functions at rates determined by the most difficult function in the composition. As an application, we derive improved convergence rates for empirical risk minimization in nonparametric regression that adapt to smoothness, compositional structure, and intrinsic dimensionality.

Optimal neural network approximation of smooth compositional functions on sets with low intrinsic dimension

TL;DR

This work analyzes deep ReLU networks under structural assumptions that mitigate dimensionality challenges. It proves minimax-optimal uniform approximation rates for -Hölder smooth functions on sets with low Minkowski dimension via networks with flexible width and depth, and introduces a novel memorization result enabling efficient point fitting in dense architectures. It then defines a compositional model where each component acts on a low-dimensional domain and shows that deep nets can approximate such functions with rates governed by the most difficult component, unifying structure and dimension. The results yield improved convergence rates for empirical risk minimization in nonparametric regression, adapting to smoothness, compositional structure, and intrinsic dimensionality, and providing a potential theoretical basis for the practical success of deep networks on high-dimensional tasks with latent low-dimensional structure.

Abstract

We study approximation and statistical learning properties of deep ReLU networks under structural assumptions that mitigate the curse of dimensionality. We prove minimax-optimal uniform approximation rates for -Hölder smooth functions defined on sets with low Minkowski dimension using fully connected networks with flexible width and depth, improving existing results by logarithmic factors even in classical full-dimensional settings. A key technical ingredient is a new memorization result for deep ReLU networks that enables efficient point fitting with dense architectures. We further introduce a class of compositional models in which each component function is smooth and acts on a domain of low intrinsic dimension. This framework unifies two common assumptions in the statistical learning literature, structural constraints on the target function and low dimensionality of the covariates, within a single model. We show that deep networks can approximate such functions at rates determined by the most difficult function in the composition. As an application, we derive improved convergence rates for empirical risk minimization in nonparametric regression that adapt to smoothness, compositional structure, and intrinsic dimensionality.
Paper Structure (20 sections, 20 theorems, 102 equations, 1 table)

This paper contains 20 sections, 20 theorems, 102 equations, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Approximation of smooth functions on sets with low Minkowski dimension
  4. Main result
  5. Optimality
  6. Outline and main tools for the proof of Theorem \ref{['thm:main-approx']}
  7. Compositional models with low Minkowski dimension
  8. Application to nonparametric regression
  9. Discussion
  10. Proofs of the main results
  11. Proof of \ref{['thm:main-approx']}
  12. Proof of \ref{['prop:optimality']}
  13. Proof of \ref{['prop:optimality-bounded']}
  14. Proof of \ref{['prop:point-fitting-ours']}
  15. Proof of \ref{['thm:main-compositional']}
  16. ...and 5 more sections

Key Result

Theorem 3.1

Let $\varepsilon > 0$, $S = [a_j, b_j]_{j = 1}^D$, $f \in \mathcal{H}_s(S, C)$, $\mathcal{M} \subseteq S$ be a set with covering number $\mathcal{N}(\varepsilon^{1/s}, \mathcal{M}, \|\cdot\|_{\infty}) \lesssim \varepsilon^{-d/s}$. Then for $N, L \in \mathds{N}$ with $N^2L^2 \asymp \varepsilon^{-d/s} In particular, if $\dim_M(\mathcal{M}) = d$, it holds $\sup_{\bm{x} \in \mathcal{M}} |f(\bm{x}) - \

Theorems & Definitions (28)

  • Definition 2.1: Class of fully connected neural networks
  • Definition 2.2: Class of Hölder smooth functions
  • Definition 2.3: Covering number
  • Definition 2.4: Minkowski dimension
  • Theorem 3.1
  • Proposition 3.2
  • Proposition 3.3
  • Proposition 3.4
  • Definition 4.1: Class of compositional models on sets with low Minkowski dimension
  • Remark 4.2
  • ...and 18 more