Optimal Approximation of Zonoids and Uniform Approximation by Shallow Neural Networks
Jonathan W. Siegel
TL;DR
The paper resolves two linked approximation problems: first, determining the Hausdorff-approximation rate of a zonoid by an $n$-summand zonotope in $\mathbb{R}^{d+1}$, and second, establishing sharp uniform approximation rates for shallow ReLU$^k$ networks on their variation spaces $\mathcal{K}_1(\mathbb{P}_k^d)$. By leveraging a dual formulation for zonoids and discrepancy/covering techniques, it removes logarithmic factors in low dimensions ($d=2,3$) and proves $\epsilon(n) \le C(d)n^{-{\tfrac{1}{2}-\tfrac{3}{2d}}}$ in all dimensions, matching the lower bounds up to constants. For neural networks, the authors derive uniform rates $\|f-f_n\|_{W^{m}(L_\infty(\Omega))} \le C\|f\|_{\mathcal{K}_1(\mathbb{P}_k^d)} n^{-{\tfrac{1}{2}-\tfrac{2(k-m)+1}{2d}}}$ for $0\le m\le k$, improving prior $L_\infty$-rates (and enabling derivative approximation) and generalizing to bounded domains. These results advance non-linear dictionary approximation theory and provide dimension-aware, derivative-inclusive rates for shallow ReLU$^k$ networks.
Abstract
We study the following two related problems. The first is to determine to what error an arbitrary zonoid in $\mathbb{R}^{d+1}$ can be approximated in the Hausdorff distance by a sum of $n$ line segments. The second is to determine optimal approximation rates in the uniform norm for shallow ReLU$^k$ neural networks on their variation spaces. The first of these problems has been solved for $d\neq 2,3$, but when $d=2,3$ a logarithmic gap between the best upper and lower bounds remains. We close this gap, which completes the solution in all dimensions. For the second problem, our techniques significantly improve upon existing approximation rates when $k\geq 1$, and enable uniform approximation of both the target function and its derivatives.