Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the expressivity of sparse maxout networks

Moritz Grillo, Tobias Hofmann

TL;DR

The paper addresses the expressivity of sparse indegree-$d$ maxout networks with multi-argument activations by establishing a duality to virtual polytopes and deriving a tight dimension bound $m_\ell$ for the associated polytopes. This geometric perspective yields a depth-based hierarchy: sufficiently deep sparse networks are universal for CPWL functions, while fixed-depth sparsity cannot be compensated by width. In the canonical case $d=r=2$, the authors fully characterize the function class as $\mathcal{N}_n(\ell,2,2)=\mathcal{M}_n(2^\ell)$, linking depth to the number of max terms and exposing exponential growth in expressivity with depth. The results are grounded in polyhedral geometry and Newton polytopes, clarifying fundamental limits of sparsity and informing depth-vs-width design in practical sparse architectures.

Abstract

We study the expressivity of sparse maxout networks, where each neuron takes a fixed number of inputs from the previous layer and employs a, possibly multi-argument, maxout activation. This setting captures key characteristics of convolutional or graph neural networks. We establish a duality between functions computable by such networks and a class of virtual polytopes, linking their geometry to questions of network expressivity. In particular, we derive a tight bound on the dimension of the associated polytopes, which serves as the central tool for our analysis. Building on this, we construct a sequence of depth hierarchies. While sufficiently deep sparse maxout networks are universal, we prove that if the required depth is not reached, width alone cannot compensate for the sparsity of a fixed indegree constraint.

On the expressivity of sparse maxout networks

TL;DR

The paper addresses the expressivity of sparse indegree- maxout networks with multi-argument activations by establishing a duality to virtual polytopes and deriving a tight dimension bound for the associated polytopes. This geometric perspective yields a depth-based hierarchy: sufficiently deep sparse networks are universal for CPWL functions, while fixed-depth sparsity cannot be compensated by width. In the canonical case , the authors fully characterize the function class as , linking depth to the number of max terms and exposing exponential growth in expressivity with depth. The results are grounded in polyhedral geometry and Newton polytopes, clarifying fundamental limits of sparsity and informing depth-vs-width design in practical sparse architectures.

Abstract

We study the expressivity of sparse maxout networks, where each neuron takes a fixed number of inputs from the previous layer and employs a, possibly multi-argument, maxout activation. This setting captures key characteristics of convolutional or graph neural networks. We establish a duality between functions computable by such networks and a class of virtual polytopes, linking their geometry to questions of network expressivity. In particular, we derive a tight bound on the dimension of the associated polytopes, which serves as the central tool for our analysis. Building on this, we construct a sequence of depth hierarchies. While sufficiently deep sparse maxout networks are universal, we prove that if the required depth is not reached, width alone cannot compensate for the sparsity of a fixed indegree constraint.
Paper Structure (7 sections, 18 theorems, 53 equations, 1 figure)

This paper contains 7 sections, 18 theorems, 53 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Further Related Work
  4. Sparse networks and continuous piecewise linear functions
  5. Virtual polytopes and support functions
  6. The dimension of polytopes arising from sparse networks
  7. Consequences for the expressivity of sparse networks

Key Result

Theorem 1

Every function $f\in\mathcal{C}_n$ can be written as for suitable $p\in\mathbb{N}$, $\beta_i\in\mathbb{R}$, and $a_{i,j}\in\mathbb{R}^n$, $i\in[p]$, $j\in[n+1]$.

Figures (1)

  • Figure 1: A network from the class $\mathcal{N}_{n}({\ell,\textbf{2},\textbf{2}})$

Theorems & Definitions (32)

  • Theorem 1: wang2005generalization
  • Theorem 2: C.f. arora2018understanding
  • proof
  • Theorem 3: schneider2013convex
  • Lemma 4: melzer2009expressibility
  • Lemma 5
  • proof
  • Definition 6
  • Theorem 7
  • proof
  • ...and 22 more