Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks

Konstantin Yakovlev, Nikita Puchkin

TL;DR

This work develops a constructive Sobolev-accurate approximation theory for simultaneously approximating the score function $s^*(y)=\nabla \log \mathsf p^*(y)$ and its derivatives up to order $m$ with GELU neural networks, under a data model $X_0=g^*(U)+\sigma Z$ that exhibits a low intrinsic dimension $d$ but lives in high ambient dimension $D$ with Gaussian noise. The authors introduce a novel, multi-step constructive proof that combines local polynomial approximations, low-dimensional function representations, and differentiable operations (exponentials, divisions, products) implemented via GELU nets, together with a partition-of-unity and tail clipping to control behavior at infinity. The main theorem delivers explicit $L^2(\mathsf p^*)$-rates for the score and all derivatives up to order $m$, along with sharp Sobolev bounds for the approximant and detailed architectural guarantees: depth $L$ that scales like $\log(mD\sigma^{-2}\log(1/\varepsilon))$, and sparsity/size bounds increasing polynomially in $D$, $d$, and the smoothness parameters, thereby avoiding the curse of dimensionality. Relative to prior work requiring bounded support, this result accommodates unbounded distributions and extends guarantees to high-order derivatives, enabling more reliable score-based diffusion modeling and density ridge estimation in noisy, high-dimensional settings.

Abstract

We present a theory for simultaneous approximation of the score function and its derivatives, enabling the handling of data distributions with low-dimensional structure and unbounded support. Our approximation error bounds match those in the literature while relying on assumptions that relax the usual bounded support requirement. Crucially, our bounds are free from the curse of dimensionality. Moreover, we establish approximation guarantees for derivatives of any prescribed order, extending beyond the commonly considered first-order setting.

Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks

TL;DR

This work develops a constructive Sobolev-accurate approximation theory for simultaneously approximating the score function and its derivatives up to order with GELU neural networks, under a data model that exhibits a low intrinsic dimension but lives in high ambient dimension with Gaussian noise. The authors introduce a novel, multi-step constructive proof that combines local polynomial approximations, low-dimensional function representations, and differentiable operations (exponentials, divisions, products) implemented via GELU nets, together with a partition-of-unity and tail clipping to control behavior at infinity. The main theorem delivers explicit -rates for the score and all derivatives up to order , along with sharp Sobolev bounds for the approximant and detailed architectural guarantees: depth that scales like , and sparsity/size bounds increasing polynomially in , , and the smoothness parameters, thereby avoiding the curse of dimensionality. Relative to prior work requiring bounded support, this result accommodates unbounded distributions and extends guarantees to high-order derivatives, enabling more reliable score-based diffusion modeling and density ridge estimation in noisy, high-dimensional settings.

Abstract

We present a theory for simultaneous approximation of the score function and its derivatives, enabling the handling of data distributions with low-dimensional structure and unbounded support. Our approximation error bounds match those in the literature while relying on assumptions that relax the usual bounded support requirement. Crucially, our bounds are free from the curse of dimensionality. Moreover, we establish approximation guarantees for derivatives of any prescribed order, extending beyond the commonly considered first-order setting.
Paper Structure (13 sections, 23 theorems, 239 equations)

This paper contains 13 sections, 23 theorems, 239 equations.

Table of Contents

  1. Introduction
  2. Preliminaries and notations
  3. Main results
  4. Proof of Theorem \ref{['thm:approx_main']}
  5. Proofs of the auxiliary results
  6. Proof of Lemma \ref{['lem:f_circ_f_star_acc']}
  7. Proof of Lemma \ref{['lem:exp_V_j_0_approx']}
  8. Proof of Lemma \ref{['lem:Upsilon_j_gelu_approx']}
  9. Proof of Lemma \ref{['lem:div_num_denum_gelu_approx']}
  10. Proof of Lemma \ref{['lem:p_star_tail']}
  11. Proof of Lemma \ref{['lem:Psi_gelu_approx']}
  12. Approximation properties of GELU neural networks
  13. Auxiliary results

Key Result

Theorem 3.2

Grant Assumption asn:relax_man. Also assume that $\varepsilon \in (0, 1)$ is sufficiently small in the sense that it satisfies and where $C_1$ and $C_2$ are absolute positive constants. Then for any $m \in \mathbb N$ there exists a score function approximation $\bar{s} \in {\mathcal{S}}(L, W, S, B)$ of the form which satisfies Moreover, $\bar{f}$ has the following configuration: where $P(d, \

Theorems & Definitions (25)

  • Definition 2.1: Sobolev space
  • Definition 2.2: Hölder class
  • Theorem 3.2: approximation of the true score function
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4
  • Lemma 4.5
  • Lemma A.1
  • Lemma B.1: approximation of identity operation with multiple layers (yakovlev2025gelu, Lemma 3.2)
  • ...and 15 more