Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dense Neural Networks are not Universal Approximators

Levi Rauchwerger, Stefanie Jegelka, Ron Levie

TL;DR

This work addresses whether densely connected neural networks can universally approximate all Lipschitz functions when depth is fixed and weights are bounded. By reinterpreting dense networks as graph neural networks through computational kernels and applying a weak regularity lemma, the authors prove a deterministic compression bound: any dense ReLU network can be approximated by a bounded-size network in the computational cut distance, implying a fundamental expressivity ceiling. They connect this compression to VC-dimension-based lower bounds, showing that for sufficiently large input dimension, dense networks cannot achieve universal approximation of Lip$(d_0,d_L)$, thus highlighting a saturation phenomenon. The results motivate sparse connectivity as a necessary ingredient for true universality and provide experiments on MNIST that illustrate the predicted saturation. Overall, the paper offers a principled compression-based lens to understand expressivity limits in dense architectures and suggests directions for analyzing more structured networks.”

Abstract

We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions if there are no restrictions on the weight values, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.

Dense Neural Networks are not Universal Approximators

TL;DR

This work addresses whether densely connected neural networks can universally approximate all Lipschitz functions when depth is fixed and weights are bounded. By reinterpreting dense networks as graph neural networks through computational kernels and applying a weak regularity lemma, the authors prove a deterministic compression bound: any dense ReLU network can be approximated by a bounded-size network in the computational cut distance, implying a fundamental expressivity ceiling. They connect this compression to VC-dimension-based lower bounds, showing that for sufficiently large input dimension, dense networks cannot achieve universal approximation of Lip, thus highlighting a saturation phenomenon. The results motivate sparse connectivity as a necessary ingredient for true universality and provide experiments on MNIST that illustrate the predicted saturation. Overall, the paper offers a principled compression-based lens to understand expressivity limits in dense architectures and suggests directions for analyzing more structured networks.”

Abstract

We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions if there are no restrictions on the weight values, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.
Paper Structure (55 sections, 23 theorems, 124 equations, 1 figure, 2 tables)

This paper contains 55 sections, 23 theorems, 124 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Graphs.
  4. Partitions.
  5. Kernels.
  6. Graph-Induced Kernels.
  7. Cut Norm.
  8. Neural Networks.
  9. Forward Propagation.
  10. $B$-Strongly Dense Networks.
  11. Graph and Kernel Neural Networks.
  12. $B$-IR-MPNN.
  13. Functions and Approximation.
  14. Universal Approximation.
  15. Network Complexity and the VC Dimension.
  16. ...and 40 more sections

Key Result

Lemma 3

Let $(\mathbf{W},\mathbf{b})$ be the dense parameters of $\Theta_{(\mathbf{W},\mathbf{b})}\in\mathcal{NN}_{\mathrm{dense}}(B,L,d_0,d_L)$. Then, for any input $\mathbf{x} \in \mathbb{R}^{d_0}$ output channel $i\in[d_L]$, we have whenever $v$ is in the $i$th interval of $\mathcal{C}^{\mathrm{out}}_{d_L}$.

Figures (1)

  • Figure 1: A computational kernel-signal with hidden dimension $d=8$.

Theorems & Definitions (39)

  • Definition 1: $B$-Strongly Dense Network
  • Definition 2
  • Remark 3
  • Lemma 3
  • Theorem 4: Computational Cut Distance Lipschitz Continuity
  • Theorem 5
  • Corollary 5: Network Architecture Approximation
  • Theorem 6
  • Theorem 7: Expressivity Bound of Strongly Dense Neural Networks
  • Lemma 8
  • ...and 29 more