Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

Ming-Jun Lai, Zhaiming Shen

TL;DR

This work proposes a principled route to overcoming the curse of dimensionality in multivariate function approximation by leveraging Kolmogorov's superposition theorem (KST) and introducing Kolmogorov-Lipschitz (KL) continuity. It shows that KL functions admit an efficient two-hidden-layer ReLU network (the K-network) with a dimension-dependent but rate-bounded $O(1/n)$ approximation using $(6d+2)n$ parameters, and then builds practical bases (KB-splines and their smoothed version, LKB-splines) to realize KST-based approximations. A key contribution is the numerical demonstration that, via discrete least-squares with a pivotal data set of size $O(nd)$ and matrix cross-approximation, one can achieve RMSE comparable to using the full $O(n^d)$ grid, effectively breaking COD for KL-like functions and enabling dimension-reduction perspectives in high-dimensional approximation. The framework combines a rigorous representation (KST, KL/KH classes) with practical spline-based bases and data-sampling strategies, offering scalable tools for high-dimensional function approximation with potential impact on dimension-reduction tasks and data-efficient learning. The results are complemented by numerical evidence in 2D/3D and discussion of extending to higher dimensions, along with a computational strategy to identify pivotal data locations that decouple data volume from model capacity.

Abstract

We explain how to use Kolmogorov Superposition Theorem (KST) to break the curse of dimensionality when approximating a dense class of multivariate continuous functions. We first show that there is a class of functions called Kolmogorov-Lipschitz (KL) continuous in $C([0,1]^d)$ which can be approximated by a special ReLU neural network of two hidden layers with a dimension independent approximation rate $O(1/n)$ with approximation constant increasing quadratically in $d$. The number of parameters used in such neural network approximation equals to $(6d+2)n$. Next we introduce KB-splines by using linear B-splines to replace the outer function and smooth the KB-splines to have the so-called LKB-splines as the basis for approximation. Our numerical evidence shows that the curse of dimensionality is broken in the following sense: When using the standard discrete least squares (DLS) method to approximate a continuous function, there exists a pivotal set of points in $[0,1]^d$ with size at most $O(nd)$ such that the rooted mean squares error (RMSE) from the DLS based on the pivotal set is similar to the RMSE of the DLS based on the original set with size $O(n^d)$. The pivotal point set is chosen by using matrix cross approximation technique and the number of LKB-splines used for approximation is the same as the size of the pivotal data set. Therefore, we do not need too many basis functions nor too many function values to approximate a high dimensional continuous function $f$. Hence, the study in this paper provides an approach for dimension reduction problems.

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

TL;DR

This work proposes a principled route to overcoming the curse of dimensionality in multivariate function approximation by leveraging Kolmogorov's superposition theorem (KST) and introducing Kolmogorov-Lipschitz (KL) continuity. It shows that KL functions admit an efficient two-hidden-layer ReLU network (the K-network) with a dimension-dependent but rate-bounded approximation using parameters, and then builds practical bases (KB-splines and their smoothed version, LKB-splines) to realize KST-based approximations. A key contribution is the numerical demonstration that, via discrete least-squares with a pivotal data set of size and matrix cross-approximation, one can achieve RMSE comparable to using the full grid, effectively breaking COD for KL-like functions and enabling dimension-reduction perspectives in high-dimensional approximation. The framework combines a rigorous representation (KST, KL/KH classes) with practical spline-based bases and data-sampling strategies, offering scalable tools for high-dimensional function approximation with potential impact on dimension-reduction tasks and data-efficient learning. The results are complemented by numerical evidence in 2D/3D and discussion of extending to higher dimensions, along with a computational strategy to identify pivotal data locations that decouple data volume from model capacity.

Abstract

We explain how to use Kolmogorov Superposition Theorem (KST) to break the curse of dimensionality when approximating a dense class of multivariate continuous functions. We first show that there is a class of functions called Kolmogorov-Lipschitz (KL) continuous in which can be approximated by a special ReLU neural network of two hidden layers with a dimension independent approximation rate with approximation constant increasing quadratically in . The number of parameters used in such neural network approximation equals to . Next we introduce KB-splines by using linear B-splines to replace the outer function and smooth the KB-splines to have the so-called LKB-splines as the basis for approximation. Our numerical evidence shows that the curse of dimensionality is broken in the following sense: When using the standard discrete least squares (DLS) method to approximate a continuous function, there exists a pivotal set of points in with size at most such that the rooted mean squares error (RMSE) from the DLS based on the pivotal set is similar to the RMSE of the DLS based on the original set with size . The pivotal point set is chosen by using matrix cross approximation technique and the number of LKB-splines used for approximation is the same as the size of the pivotal data set. Therefore, we do not need too many basis functions nor too many function values to approximate a high dimensional continuous function . Hence, the study in this paper provides an approach for dimension reduction problems.
Paper Structure (12 sections, 22 theorems, 62 equations, 7 figures, 2 tables)

This paper contains 12 sections, 22 theorems, 62 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Universal Approximation via KST
  3. Kolmogorov-Lipschitz (KL) Functions
  4. Approximation via ReLU Neural Networks
  5. Functions Beyond the KL Class
  6. KB-splines and LKB-splines
  7. KB-splines
  8. LKB-splines
  9. Numerical Approximation by LKB-splines
  10. The LKB-splines are Dense in $C([0, 1]^d)$
  11. The pivotal data locations for breaking the curse of dimensionality
  12. Numerical Approximation by LKB-splines from a Dimension Reduction Point of View

Key Result

Theorem 1

For every function $f$ in $\Gamma_{B,C}$, every sigmoidal function $\phi$, every probability measure $\mu$, and every $n \ge 1$, there exists a linear combination of sigmoidal functions $f_n(x)$, a shallow neural network, such that The coefficients of the linear combination in $f_n$ may be restricted to satisfy $\sum_{k=1}^n |c_k|\le 2C$, and $c_0 = f(0)$.

Figures (7)

  • Figure 1: Original image (left column), reconstructed image (middle column), and associated outer function $g$ (right column)
  • Figure 2: Top left: reconstruction of $f(x,y)=x$. Top right: reconstruction of $f(x,y)=x^2$. Bottom left: reconstruction of $f(x,y)=\cos(2(x-y)/\pi)$. Bottom right: reconstruction of $f(x,y)=\sin(1/(1+(x-0.5)(y-0.5)))$.
  • Figure 4: Examples of LKB-splines (first and third columns) which are the smoothed version of the corresponding KB-splines (second and fourth columns).
  • Figure 5: The sparsity pattern of data matrix for $n=1000$.
  • Figure 6: Pivotal data at 99 locations (selected from $41^2$ equally-spaced locations) in 2D and 178 locations (selected from $41^3$ equally-spaced locations) in 3D for $n=100$.
  • ...and 2 more figures

Theorems & Definitions (28)

  • Theorem 1: Barron, 1993
  • Definition 1
  • Theorem 2: Kolmogorov Superposition Theorem
  • Theorem 3: Universal Approximation Theorem (cf. SM15
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • ...and 18 more