Neural Network Approximation: A View from Polytope Decomposition

TL;DR

The paper develops a polytope-decomposition framework for universal approximation with ReLU networks, arguing that real networks partition space into polytopes rather than cubes. It introduces a kernel polynomial method (KPM) combined with the Totik-Ditzian modulus to achieve explicit, adaptive approximations on polytopes, and provides a constructive network realization with explicit width and depth bounds. For continuous target functions on a polytope $K$, it proves a universal-approximation theorem with error governed by $\widetilde{\omega}_K(f,N^{-1})+N^{-\alpha}$ and constructs networks with width $O\big(\max\{N^3,N^{d-1}\log N\}\big)$ and depth $O(\log N)$, vanishing outside $K$. It also extends the approach to analytic functions, achieving exponential rates, and discusses adaptability to local regularity and manifold structures, highlighting practical implications for task-aware network design and learnability.

Abstract

Universal approximation theory offers a foundational framework to verify neural network expressiveness, enabling principled utilization in real-world applications. However, most existing theoretical constructions are established by uniformly dividing the input space into tiny hypercubes without considering the local regularity of the target function. In this work, we investigate the universal approximation capabilities of ReLU networks from a view of polytope decomposition, which offers a more realistic and task-oriented approach compared to current methods. To achieve this, we develop an explicit kernel polynomial method to derive an universal approximation of continuous functions, which is characterized not only by the refined Totik-Ditzian-type modulus of continuity, but also by polytopical domain decomposition. Then, a ReLU network is constructed to approximate the kernel polynomial in each subdomain separately. Furthermore, we find that polytope decomposition makes our approximation more efficient and flexible than existing methods in many cases, especially near singular points of the objective function. Lastly, we extend our approach to analytic functions to reach a higher approximation rate.

Figures (7)

• Figure 1: While universal approximation theories partition the space into hypercubes, a ReLU network actually divides the space into polytopes.
• Figure 1: (a) A classical ReLU network of widths $3$ and $2$ for two hidden layers. (b) Intra-linked ReLU network of width $3$ and $2$ for two hidden layers.
• Figure 1: An illustration of constructing sawtooth functions in \ref{['Sawtooth']}.
• Figure 1: An illustration of constructing $\widetilde{f}_k$ by ReLU networks.
• Figure 2: Illustration of the Ditzian-Totik modulus of continuity. (a) The modulus of continuity of $f:[-1,1]\to \mathbb{R}$ with step $t$ is computed by taking supremum of $f(x+\frac{hw(x)}{2})-f(x-\frac{hw(x)}{2})$ over $0<h<t$ and $-1\leq x\leq 1$, where $w(x)$ is a weight function measuring the distance from $x$ to the boundary. When $w(x)\equiv1$ and $w(x)=\phi(x)=\sqrt{1-x^2}$, we obtain the ordinary and Ditzian-Totik modulus of continuity respectively. (b) $\boldsymbol{e}_{i},i=1,2,3,4$ are the direction of edges of polytope $K$. For $\boldsymbol{x}\in K$, $l_{\boldsymbol{e_i},\boldsymbol{x}}$ is a straight line through $\boldsymbol{x}$ and parallel to $\boldsymbol{e}_i$ intersecting $K$ at $\boldsymbol{a}_{\boldsymbol{e}_i,\boldsymbol{x}}$ and $\boldsymbol{b}_{\boldsymbol{e}_i,\boldsymbol{x}}$.

Theorems & Definitions (46)

• Theorem 1.1
• Theorem 1.2
• Example 1.3
• Proof 1
• Example 1.4
• Remark 1.5
• Definition 2.1: Classical ReLU network
• Definition 2.2: Intra-linked ReLU network
• Theorem 2.3
• Theorem 2.4