Fast and Exact Enumeration of Deep Networks Partitions Regions

TL;DR

This work tackles the problem of characterizing the input-space partition $Ω$ induced by continuous piecewise affine (CPA) deep networks, a task previously limited to low-dimensional slices or sampling. It introduces a parallel, exact enumeration method for single-layer partitions based on hyperplane arrangements and a reverse-search algorithm, then extends the approach to multi-layer nets via recursive layer-wise subdivision, leveraging the CPA property that preserves linearity within regions. The authors demonstrate that uniform sampling is exponentially inefficient for discovering small regions while their method scales linearly with the input dimension and region count, making it a practical baseline for exact partition computation and for validating CPA-based analyses. This enumeration capability enables precise assessment of approximation techniques, improves neural architecture search (NAS) pipelines, and provides a principled tool for exploring the geometric structure of deep networks.

Abstract

One fruitful formulation of Deep Networks (DNs) enabling their theoretical study and providing practical guidelines to practitioners relies on Piecewise Affine Splines. In that realm, a DN's input-mapping is expressed as per-region affine mapping where those regions are implicitly determined by the model's architecture and form a partition of their input space. That partition -- which is involved in all the results spanned from this line of research -- has so far only been computed on $2/3$-dimensional slices of the DN's input space or estimated by random sampling. In this paper, we provide the first parallel algorithm that does exact enumeration of the DN's partition regions. The proposed algorithm enables one to finally assess the closeness of the commonly employed approximations methods, e.g. based on random sampling of the DN input space. One of our key finding is that if one is only interested in regions with ``large'' volume, then uniform sampling of the space is highly efficient, but that if one is also interested in discovering the ``small'' regions of the partition, then uniform sampling is exponentially costly with the DN's input space dimension. On the other hand, our proposed method has complexity scaling linearly with input dimension and the number of regions.

Figures (2)

• Figure 1: Proposed exact region enumeration depicted as an orange staragainst sampling-based region discovery of the partition $\Omega$ depicted as blue dotsfor a single hidden layer DN with leaky-ReLU, random parameters and width $64$ as a function of computation time ( x-axis) and number of partition regions found ( y-axis); for a $4$-dimensional input space at the top and $8$-dimensional input space at the bottom. The proposed \ref{['algo:police']} is able to dramatically outperform the sampling-based search that has been used throughout recent studies on CPA DNs.
• Figure 2: Depiction of the multilayer case which corresponds to a union of region-constrained hyperplane arrangements and thus which can be studied directly form the proposed hyperplane arrangement region enumeration. The only additional step is to first enforce that the search takes place on the restricted region of interest from the up-to-layer-$\ell$ input space partition. For example on the left column one first obtains the first layer partition depicted in black. On each of the enumerated region, a subdivision will be performed by the next layer; pick any region of interest, compose the per-region affine mapping (fixed on that region) with the second layer affine mappings, and repeat the region enumeration algorithm. This discovers the second subdivision done by the second layer, highlighted in blue in the middle column. This can be repeated to obtain the subdivision of the third layer, here highlighted in red in the right column.