Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Algorithms for Exact Enumeration of Deep Neural Network Activation Regions

Sabrina Drammis, Bowen Zheng, Karthik Srinivasan, Robert C. Berwick, Nancy A. Lynch, Robert Ajemian

TL;DR

This work tackles the exact enumeration of activation regions in ReLU-based neural networks by introducing a parallel framework, LayerWise-NNCE, and a concrete algorithm, ParLayerWise1-NNCE, for scalable region enumeration. It formalizes the NN-CellEnum problem, analyzes two problem settings, and demonstrates that parallelism is crucial for handling networks with hundreds of millions of regions. Empirical results on a two-hidden-layer network trained on EMNIST reveal linear scaling between region count and runtime, while providing insights into how region dimensions drive deeper partitioning and affect network performance. The study thereby offers both methodological tools for exact region analysis and practical findings about the organization of activation regions across layers, with implications for network design and interpretability.

Abstract

A feedforward neural network using rectified linear units constructs a mapping from inputs to outputs by partitioning its input space into a set of convex regions where points within a region share a single affine transformation. In order to understand how neural networks work, when and why they fail, and how they compare to biological intelligence, we need to understand the organization and formation of these regions. Step one is to design and implement algorithms for exact region enumeration in networks beyond toy examples. In this work, we present parallel algorithms for exact enumeration in deep (and shallow) neural networks. Our work has three main contributions: (1) we present a novel algorithm framework and parallel algorithms for region enumeration; (2) we implement one of our algorithms on a variety of network architectures and experimentally show how the number of regions dictates runtime; and (3) we show, using our algorithm's output, how the dimension of a region's affine transformation impacts further partitioning of the region by deeper layers. To our knowledge, we run our implemented algorithm on networks larger than all of the networks used in the existing region enumeration literature. Further, we experimentally demonstrate the importance of parallelism for region enumeration of any reasonably sized network.

Parallel Algorithms for Exact Enumeration of Deep Neural Network Activation Regions

TL;DR

This work tackles the exact enumeration of activation regions in ReLU-based neural networks by introducing a parallel framework, LayerWise-NNCE, and a concrete algorithm, ParLayerWise1-NNCE, for scalable region enumeration. It formalizes the NN-CellEnum problem, analyzes two problem settings, and demonstrates that parallelism is crucial for handling networks with hundreds of millions of regions. Empirical results on a two-hidden-layer network trained on EMNIST reveal linear scaling between region count and runtime, while providing insights into how region dimensions drive deeper partitioning and affect network performance. The study thereby offers both methodological tools for exact region analysis and practical findings about the organization of activation regions across layers, with implications for network design and interpretability.

Abstract

A feedforward neural network using rectified linear units constructs a mapping from inputs to outputs by partitioning its input space into a set of convex regions where points within a region share a single affine transformation. In order to understand how neural networks work, when and why they fail, and how they compare to biological intelligence, we need to understand the organization and formation of these regions. Step one is to design and implement algorithms for exact region enumeration in networks beyond toy examples. In this work, we present parallel algorithms for exact enumeration in deep (and shallow) neural networks. Our work has three main contributions: (1) we present a novel algorithm framework and parallel algorithms for region enumeration; (2) we implement one of our algorithms on a variety of network architectures and experimentally show how the number of regions dictates runtime; and (3) we show, using our algorithm's output, how the dimension of a region's affine transformation impacts further partitioning of the region by deeper layers. To our knowledge, we run our implemented algorithm on networks larger than all of the networks used in the existing region enumeration literature. Further, we experimentally demonstrate the importance of parallelism for region enumeration of any reasonably sized network.
Paper Structure (20 sections, 1 theorem, 5 equations, 8 figures, 1 table, 5 algorithms)

This paper contains 20 sections, 1 theorem, 5 equations, 8 figures, 1 table, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Background
  3. Deep Neural Networks
  4. Hyperplane Arrangements
  5. Cell Partitioning in Shallow Neural Networks
  6. Cell Partitioning in Deep Neural Networks
  7. Problem statement
  8. A framework for neural network cell enumeration algorithms
  9. An Algorithm Framework for the NN-CellEnum Problem
  10. Existing Serial Algorithms for Cell Enumeration of Hyperplane Arrangements
  11. Parallel algorithms for the neural network cell enumeration problem
  12. ParLayerWise1-NNCE: An Embarrassingly Parallel Algorithm for Problem Setting 1
  13. ParLayerWise1-NNCE Correctness Proof and Performance Analysis
  14. Parallel Algorithms for Problem Setting 2
  15. Implementation, Experiments, and Results
  16. ...and 5 more sections

Key Result

Theorem 1

Assume we are in problem setting 1: $n_1 \leq n_0$ and thus $|\mathcal{C}^1| \approx 2^{n_1}$. The total expected work and span of Algorithm alg:setting1 are $O(L n_{max} \textrm{lp}(*) |\mathcal{C}^L|)$ and $O(\zeta L n_{max} \textrm{lp}(*) |\mathcal{C}^L|)$, respectively -- where $|\mathcal{C}^L|$

Figures (8)

  • Figure 1: An example neural network (left top), its partitioning of the bounded domain $\mathbb{X}$ into cells (right), and the activation pattern, $Z$, and network sign vector, $V$, corresponding to $\textbf{p} \in \mathbb{X}$ (left bottom).
  • Figure 2: Sample images from our 15 class subset of the EMNIST dataset.
  • Figure 3: ParLayerWise1-NNCE (Algorithm \ref{['alg:setting1']}) performance (top) and individual task time for the example network outlined in light blue (bottom).
  • Figure 4: Serial versus parallel runtime for all two-layer networks with architecture $n_1 = n_2 = 11$.
  • Figure 5: Number of activation regions versus classification accuracy for all networks from Figure \ref{['fig:alg-performance']}.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Definition 2.1: Activation pattern
  • Definition 2.2: Activation region
  • Definition 2.3: Hyperplane arrangement
  • Definition 2.4: Cell
  • Definition 2.5: Sign vector
  • Definition 2.6: Network sign vector
  • Definition 2.7: Network sign vector prefix
  • Theorem 1