Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the expressivity of deep Heaviside networks

Insung Kong, Juntong Chen, Sophie Langer, Johannes Schmidt-Hieber

TL;DR

This work analyzes the expressivity of deep Heaviside networks (DHNs) and shows that plain DHNs are limited by their one-bit-per-neuron transmissions, but this limitation can be overcome by augmenting the architecture with skip connections or linear neurons. The authors derive tight VC-dimension bounds and approximation rates for skip-DHNs and lin-DHNs, illustrating how depth, width, and augmentation interact to control capacity and function approximation, including for Hölder-smooth targets. They further apply these results to nonparametric regression, showing that with appropriate choices of depth and width, skip-DHNs and lin-DHNs achieve the minimax rate $n^{-2β/(2β+d)}$ (up to log factors) over Hölder classes, matching known optimal rates. The discussion situates these findings relative to ReLU networks and other DHN variants, addressing training challenges and potential advantages in computation and energy efficiency, and suggests directions for practical deployment and further theoretical refinement.

Abstract

We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.

On the expressivity of deep Heaviside networks

TL;DR

This work analyzes the expressivity of deep Heaviside networks (DHNs) and shows that plain DHNs are limited by their one-bit-per-neuron transmissions, but this limitation can be overcome by augmenting the architecture with skip connections or linear neurons. The authors derive tight VC-dimension bounds and approximation rates for skip-DHNs and lin-DHNs, illustrating how depth, width, and augmentation interact to control capacity and function approximation, including for Hölder-smooth targets. They further apply these results to nonparametric regression, showing that with appropriate choices of depth and width, skip-DHNs and lin-DHNs achieve the minimax rate (up to log factors) over Hölder classes, matching known optimal rates. The discussion situates these findings relative to ReLU networks and other DHN variants, addressing training challenges and potential advantages in computation and energy efficiency, and suggests directions for practical deployment and further theoretical refinement.

Abstract

We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.
Paper Structure (28 sections, 21 theorems, 269 equations, 16 figures, 1 table)

This paper contains 28 sections, 21 theorems, 269 equations, 16 figures, 1 table.

Table of Contents

  1. Introduction
  2. Deep Heaviside networks (DHNs)
  3. Mathematical definition
  4. Limited expressivity of DHNs
  5. Skip connections augmented DHNs
  6. Mathematical definition
  7. Approximation of the square function
  8. Bounds on the VC dimension
  9. Approximation of $\beta$-Hölder smooth functions
  10. Linear neurons augmented DHNs
  11. Mathematical definition
  12. Bounds on the VC dimension
  13. Approximation of smooth functions
  14. Application to nonparametric regression
  15. Discussion
  16. ...and 13 more sections

Key Result

Theorem 1

For any $L$, any $\bm{p} = (d,p_1,\ldots,p_L,1)$, any $f \in \operatorname{DHN}(L,\bm{p})$, and any $\mathbf{x}_1, \mathbf{x}_2 \in [0,1]^d$, the function $f|_{[\mathbf{x}_1, \mathbf{x}_2]}$ is piecewise constant with at most $p_1+1$ pieces. Hence, for any function $f_0 : [0,1]^d \to \mathbb{R}$,

Figures (16)

  • Figure 1: The network architecture representing the network class $\operatorname{DHN}(L,\bm{p})$ with $L=4$ and $\bm{p}=(2,3,3,3,2,1).$
  • Figure 2: Example of skip connections augmented DHN (skip-DHN). Hidden neurons in red are directly connected to the input. In particular, the second hidden layer contains two skip connected neurons ($s_2=2$), the third hidden layer has one skip connected neuron ($s_3=1$) and the last hidden layer has none ($s_4=0$). The graph represents the network class $\operatorname{DHN}_{\operatorname{skip}}(L,\bm{p}, \bm{s})$ with $L=4$, $\bm{p}=(2,3,3,3,2,1)$ and $\bm{s} = (2,1,0)$.
  • Figure 3: Example of linear neurons augmented DHN (lin-DHN). All (except the last) hidden layers are augmented by red linear neurons. The graph represents the network class $\operatorname{DHN}_{\operatorname{lin}}(L,\bm{p}, s)$ with $L=4$, $\bm{p}=(2,3,3,3,2,1)$ and $s=1$.
  • Figure 4: Illustration for the proof of Lemma \ref{['lemma_bit_extract']}. Neurons in red squares are skip connected to the input, while those in blue squares are not.
  • Figure 5: Example of $U_j$, $V_k$ and $W_r$ in Lemma \ref{['lemma_piece_bin_skip_ver3']}. In this case, $(d,m,n)=(2,1,2)$, and hence $(J,K,R)=(4,4,16)$. The green, blue, and red solid squares represent $U_1$, $V_1$, and $W_1$, respectively. For each $j \in [J]$, $k \in [K]$ and $r \in [R]$, $U_{j} \cap V_{k} \cap W_{r}$ is a hypercube with the side length $2^{-2m-n}$.
  • ...and 11 more figures

Theorems & Definitions (46)

  • Definition 1
  • Definition 2: Piecewise constant function
  • Theorem 1
  • Proposition 2
  • Remark 1
  • Theorem 3
  • Theorem 4
  • Corollary 5
  • Definition 3: Growth function, Shattering, VC dimension
  • Theorem 6
  • ...and 36 more