The Computational Complexity of Counting Linear Regions in ReLU Neural Networks

TL;DR

An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space, and it is demonstrated that counting linear regions can at least be achieved in polynomial space for some common definitions.

Abstract

An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space. There exist many different, non-equivalent definitions of what a linear region actually is. We systematically assess which papers use which definitions and discuss how they relate to each other. We then analyze the computational complexity of counting the number of such regions for the various definitions. Generally, this turns out to be an intractable problem. We prove NP- and #P-hardness results already for networks with one hidden layer and strong hardness of approximation results for two or more hidden layers. Finally, on the algorithmic side, we demonstrate that counting linear regions can at least be achieved in polynomial space for some common definitions.

Figures (6)

• Figure 1: A ReLU network computing the function $f(x,y)=\max(-y, \min(0,-x))$. The closed connected regions (center left) and the activation regions (center right) are displayed. The slice $\{(x,0):x\geq 0\}$ is contained in the two closed connected regions with functions $0$ (red) and $-y$ (blue). We have $R_6=R_5=R_4=3$, $R_3=4$, $R_2=6$ and $R_1=8$. In anti-clockwise direction starting from the region with value 0, the activation patterns are $010110$, $000000$, $001101$, $001001$, $101001$, $100000$, $110010$ and $110110$ (neurons are ordered from the upper left to the lower right).
• Figure 2: The function $\max(0,x)+\max(0,-y) - \max(0,x-y) +\min(\max(0, x-2), \max(0, y-2)) - 2 \min(\max(0, x-1), \max(0, y-1))$. We have $R_6=7$, $R_5=8$, and $R_4=9$.
• Figure 3: The function $T_1$ (left), $T_2$ (center) and its linear regions (right).
• Figure 4: The functions $T_{1,\varepsilon}$ (left), $T_{2,\varepsilon}$ (center) and its linear regions (right).
• Figure 5: A ReLU network $N$ computing $\min(y, \max(-1, -x), \max(3 - 2 x, -x))$. An activation region of $N$ is either a blue line, blue point, or a full dimensional cell as defined by the blue lines. There are four closed connected region as indicated by the colors. The line between the points $(1,-1)$ and $(3,-3)$ belongs to the green as well as the orange region.

Theorems & Definitions (51)

• Definition 1: Activation Region
• Definition 2: Proper Activation Region
• Definition 3: Convex Region
• Definition 4: Open Connected Region
• Definition 5: Closed Connected Region
• Definition 6: Affine Region
• Theorem 3.1
• Theorem 4.1
• Theorem 4.2
• proof : Proof sketch